nilesh.p Posted November 26 Posted November 26 Hi All, We are using Azure Databricks clusters as a data source (will potentially move to Serverless SQL) for Seeq. However, we would prefer to reduce cluster uptime to business hours to save costs. It seems that Seeq continues to ping the cluster and brings it up every 30 minutes. Is there a setting or schedule that can be applied to the data sources so that it only remains active during the day?
Seeq Team James DeMarco Posted November 26 Seeq Team Posted November 26 Indeed, there is a parameter "ShouldMonitor" for Databricks connections that makes it possible to disable connection monitoring - see https://support.seeq.com/kb/latest/cloud/databricks-lakehouse#DatabricksLakehouse-Databricks-SpecificSQLAdditionalConfiguration
nilesh.p Posted November 26 Author Posted November 26 Thanks for the response. This parameter is already being used and is set to false, but the cluster is still being brought up every half hour.
Seeq Team James DeMarco Posted November 27 Seeq Team Posted November 27 Hi Nilesh, In that case, it's likely that some user-initiated automated activity is inducing the connection, e.g., a scheduled notebook querying for data from Databricks. You may be able to identify this by looking at the Requests, Jobs, or Schedules tabs in the Administration UI, or in the logs - if you aren't able to figure out the reason for the activity from there, you can file a support request at support.seeq.com for assistance from our team. Cheers, James
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now