Brighton Rust Posted May 29 Share Posted May 29 We are developing a third party Seeq add-on and we are using the SPY library to pull data. For pulling ~50 signals, it can take up to 2 minutes to pull with a grid size of 15 minutes on the demo server and several times longer on the learn server. For our final product, we will need to pull a few thousand signals at regular intervals. We are concerned that spy.pull() is too slow. Is there a different way to pull data from Seeq more quickly? Link to comment Share on other sites More sharing options...
Seeq Team Selmane Posted May 30 Seeq Team Share Posted May 30 Hello Brighton, Spy.pull() is the only method available in Seeq Data Lab to retrieve data from the data source. Speed of data retrieval is affected by multiple factors: The number of time series you are pulling The time range you are pulling data for The type of data source you are using For the tests you are doing, can you be more precise on the data source and the time range you are pulling data for? Thanks, --Selmane Link to comment Share on other sites More sharing options...
Seeq Team John Posted May 31 Seeq Team Share Posted May 31 I’d like to get some additional information about the Add-on you are looking to develop. You mention that “For our final product, we will need to pull a few thousand signals at regular intervals”. I’m curious what the use case is for this amount of signals. SPy parallelizes the pull requests and is most likely going to be your most optimal route. By default `spy.pull` pulls 8 items concurrently. This can be adjusted to a little higher number which may help a bit. You can use the following to adjust, but I don't know how much this will help. spy.options.max_concurrent_requests = 16 My other concern is related to memory allocation. During a pull, the item data is held in memory and I’m worried that if you are pulling a few thousand signals that you can run out of memory. It all depends on how many signals and time range you plan on pulling. Chunking the pull into smaller groups of signals may relieve the memory pressure. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now