Best Practices for Scheduled Python Calculations

[Shamus Cunningham]

In Seeq version 52 and beyond we have introduced scheduled notebooks which opens up a new world of signals calculated using purely python and pushed into the seeq system on a schedule. These scheduled executions can be really powerful but also can place huge loads on the entire Seeq system if your notebooks are not designed with scheduled execution in mind. Attached is a notebook with some of our current thoughts on best practices to minimize the load on the system and not request and push the same sets of data over and over again with each notebook execution. 

General Calculation Plan 

• Query our calculated signal name between now - Lookback length (1day) 
• Find the last timestamp of our calculated signal 
• Query the input signals to our calculation for the time range between that last timestamp and now
• Run our python script on our input signals to generate a new range of values for our calculated signal 
• Push results back into Seeq for range between last timestamp and now
• Schedule next execution (15 min)

 

Scenarios/methods you may want to avoid include pulling a month of data for all input signals, calculating values and then pushing results back on a 15 minute schedule. This will result in needlessly reading and writing the same data points over and over again. 

Example Scheduled Signal Calculation (1).ipynb

[VpJ]

What's the best way to upload a plot from datalab to organizr? As of now, i am just saving it as an image and pasting it in organizr. Is there an automatic way to link it to organizr? Ideally I would like to schedule once/day job in jupyterNB and update the plot in Organizr. 

 

[Sanman Mehta]

Hi VpJ,

You are on the right track. Schedule daily jobs by adding this code to your data lab project script

spy.jobs.schedule("Every 1 Day") 

In a future release you will be able to create python visuals in qorkbench and just link it to organizer topic (same as you do with standard trends, tables, treemap views).  

[SBC]

I need to calculate a simple difference (delta) between two time series signals feeding from Continuous Historian to SeeQ over the past history (10 years) and keep it available in SeeQ for other aggregate calculations. This will have to be done over several Assets in an Asset tree (similar to  the Example tree above). Currently, my workbooks are trying to calculate 'just-in-time' whenever we open the workbook which is causing a lot of time lag in getting the outputs as it is waiting to first pull and calculate the delta value between the two signals.

Will taking the above approach speed up my worksheet calculation performance/speed if the the delta value was pre-pushed with SeeQ data lab on a regular basis (say, daily) ? I am also planning to combine with the Asset tree signal push script shared by SeeQ team member in another post.

 

[Sanman Mehta]

Seeq stores all results in cache (most cases), so once you have done calculations once, they should not need recalculation for 10 years every time and resulting signal can be instantly used in further calculations.  This should help alleviate calculation load from 2nd time onwards such that only new calculation would be for the last 1 day.  You should not have to run scripts for 10 years everyday. 

Sanman

[SBC]

1 hour ago, Sanman Mehta said:

Seeq stores all results in cache (most cases), so once you have done calculations once, they should not need recalculation for 10 years every time and resulting signal can be instantly used in further calculations.  This should help alleviate calculation load from 2nd time onwards such that only new calculation would be for the last 1 day.  You should not have to run scripts for 10 years everyday. 

Sanman

Thank you Sanman  - I believe there is a difference in speed when the calculated signal is pushed as an "Asset Attribute" (and cached for each Asset) VS. keeping it as a "work-sheet  (cached formula has to re-populate when swapping assets). 

[Sanman Mehta]

SBC, unless I am misinterpreting your message, the cache should behave in same performance. Please reach out to me directly if you are experiencing slowness in any case.

Sanman