SPY data pulling speeds

[Brighton Rust]

We are developing a third party Seeq add-on and we are using the SPY library to pull data. 

 For pulling ~50 signals, it can take up to 2 minutes to pull with a grid size of 15 minutes on the demo server and several times longer on the learn server.

 For our final product, we will need to pull a few thousand signals at regular intervals. We are concerned that spy.pull() is too slow.

  Is there a different way to pull data from Seeq more quickly? 

[Selmane]

Hello Brighton,

Spy.pull() is the only method available in Seeq Data Lab to retrieve data from the data source. Speed of data retrieval is affected by multiple factors:

• The number of time series you are pulling
• The time range you are pulling data for 
• The type of data source you are using

For the tests you are doing, can you be more precise on the data source and the time range you are pulling data for?

Thanks,

--Selmane

[John]

I’d like to get some additional information about the Add-on you are looking to develop.  You mention that “For our final product, we will need to pull a few thousand signals at regular intervals”.  I’m curious what the use case is for this amount of signals.  SPy parallelizes the pull requests and is most likely going to be your most optimal route. By default `spy.pull` pulls 8 items concurrently.  This can be adjusted to a little higher number which may help a bit.  You can use the following to adjust, but I don't know how much this will help.  

spy.options.max_concurrent_requests = 16

My other concern is related to memory allocation.  During a pull, the item data is held in memory and I’m worried that if you are pulling a few thousand signals that you can run out of memory.  It all depends on how many signals and time range you plan on pulling.  Chunking the pull into smaller groups of signals may relieve the memory pressure.