Jump to content

Best Practices for Scheduled Python Calculations


Recommended Posts

  • Super Seeqer

In Seeq version 52 and beyond we have introduced scheduled notebooks which opens up a new world of signals calculated using purely python and pushed into the seeq system on a schedule. These scheduled executions can be really powerful but also can place huge loads on the entire Seeq system if your notebooks are not designed with scheduled execution in mind. Attached is a notebook with some of our current thoughts on best practices to minimize the load on the system and not request and push the same sets of data over and over again with each notebook execution. 

General Calculation Plan 

  • Query our calculated signal name between now - Lookback length (1day) 
  • Find the last timestamp of our calculated signal 
  • Query the input signals to our calculation for the time range between that last timestamp and now
  • Run our python script on our input signals to generate a new range of values for our calculated signal 
  • Push results back into Seeq for range between last timestamp and now
  • Schedule next execution (15 min)

 

Scenarios/methods you may want to avoid include pulling a month of data for all input signals, calculating values and then pushing results back on a 15 minute schedule. This will result in needlessly reading and writing the same data points over and over again. 

screenshot-explore.seeq.com-2021.07.30-15_16_25.png

Example Scheduled Signal Calculation (1).ipynb

Link to comment
Share on other sites

  • 1 year later...

What's the best way to upload a plot from datalab to organizr? As of now, i am just saving it as an image and pasting it in organizr. Is there an automatic way to link it to organizr? Ideally I would like to schedule once/day job in jupyterNB and update the plot in Organizr. 

 

Link to comment
Share on other sites

Hi VpJ,

You are on the right track. Schedule daily jobs by adding this code to your data lab project script

spy.jobs.schedule("Every 1 Day") 

In a future release you will be able to create python visuals in qorkbench and just link it to organizer topic (same as you do with standard trends, tables, treemap views).  

Link to comment
Share on other sites

I need to calculate a simple difference (delta) between two time series signals feeding from Continuous Historian to SeeQ over the past history (10 years) and keep it available in SeeQ for other aggregate calculations. This will have to be done over several Assets in an Asset tree (similar to  the Example tree above). Currently, my workbooks are trying to calculate 'just-in-time' whenever we open the workbook which is causing a lot of time lag in getting the outputs as it is waiting to first pull and calculate the delta value between the two signals.

Will taking the above approach speed up my worksheet calculation performance/speed if the the delta value was pre-pushed with SeeQ data lab on a regular basis (say, daily) ? I am also planning to combine with the Asset tree signal push script shared by SeeQ team member in another post.

 

Link to comment
Share on other sites

Seeq stores all results in cache (most cases), so once you have done calculations once, they should not need recalculation for 10 years every time and resulting signal can be instantly used in further calculations.  This should help alleviate calculation load from 2nd time onwards such that only new calculation would be for the last 1 day.  You should not have to run scripts for 10 years everyday. 

Sanman

Link to comment
Share on other sites

1 hour ago, Sanman Mehta said:

Seeq stores all results in cache (most cases), so once you have done calculations once, they should not need recalculation for 10 years every time and resulting signal can be instantly used in further calculations.  This should help alleviate calculation load from 2nd time onwards such that only new calculation would be for the last 1 day.  You should not have to run scripts for 10 years everyday. 

Sanman

Thank you Sanman  - I believe there is a difference in speed when the calculated signal is pushed as an "Asset Attribute" (and cached for each Asset) VS. keeping it as a "work-sheet  (cached formula has to re-populate when swapping assets). 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...