Advantage/Disadvantages of using spy.jobs.schedule versus something external?

[Ben Hines]

I have a simple DataLab notebook that identifies new signals and adds them to an asset tree.  It seems pretty simple to have this run periodically using spy.jobs.schedule.  Another option I'm considering is to extract the code from my notebook into a .py file and run it periodically on an existing Airflow infrastructure.  Some of the advantages of using my own infrastructure is include monitoring/alerting, log collection, etc.  Are there other aspects I should be considering?  For instance, there is a prominent warning in the spy documentation to be careful about scheduling jobs due to resource consumption and adverse effects on performance.  It's unclear how script complexity will affect performance.  I'm not even sure how to monitor performance of a scheduled notebook.  Any advice you can provide would be appreciated.  

[John]

Using Airflow to schedule your notebook is a fine approach. One thing to consider is with Seeq Data Lab, you're automatically authenticated when running a notebook, but if you move the code to Airflow, you'll need to handle authentication manually with a spy.login.  It's recommended to use an Access Key for this type of login.

Regarding performance, the warning in the spy.jobs.schedule documentation mainly refers to how frequently your script is run rather than the complexity of the Python code itself. The performance bottleneck is typically related to the data retrieval from the datasource, which will impact Seeq's performance whether you use Data Lab or Airflow. Be mindful of the frequency, and how much data, your script pulls. Running large pull requests often can degrade performance.   

In terms of monitoring, the performance of a scheduled notebook in Seeq is identical to running it manually from Data Lab. One way is to monitor the script's affect on performance would be to watch the Server Load on the home screen while the script is running.