Jump to content

Leaderboard

Popular Content

Showing content with the highest reputation since 05/23/2023 in all areas

  1. Proportional-integral-derivative (PID) control loops are essential to the automation and control of manufacturing processes and are foundational to the financial gains realized through advanced process control (APC) applications. Because poorly performing PID controllers can negatively impact production capacity, product quality, and energy consumption, implementing controller performance monitoring analytics leads to new data-driven insights and process optimization opportunities. The following sections provide the essential steps for creating basic controller performance monitoring at scale in Seeq. More advanced CLPM solutions can be implemented by expanding the standard framework outlined below with additional metrics, customization features, and visualizations. Data Lab’s Spy library functionality is integral to creating large scale CLPM, but small scale CLPM is possible with no coding, using Asset Groups in Workbench. Key steps in creating a CLPM solution include: Controller asset tree creation Developing performance metric calculations and inserting them in the tree as formulas Calculating advanced metrics via scheduled Data Lab notebooks (if needed) Configuring asset scaled and individual controller visualizations in Seeq Workbench Setting up site performance as well as individual controller monitoring in Seeq Organizer Using these steps and starting from only the controller signals within Seeq, large scale CLPM monitoring can be developed relatively quickly, and a variety of visualizations can be made available to the manufacturing team for monitoring and improving performance. As a quick example of many end result possibilities, this loop health treemap color codes controller performance (green=good, yellow=questionable, red=poor): The key steps in CLPM implementation, summarized above, are detailed below. Note: for use as templates for development of your own CLPM solution, the associated Data Lab notebooks containing the example code (for Steps 1-3) are included as file attachments to this article. The code and formulas described in Steps 1-3 can be adjusted and expanded to customize your CLPM solution as desired. Example CLPM Solution: Detailed Steps STEP 1: Controller asset tree creation An asset tree is the key ingredient which enables scaling of the performance calculations across a large number of controllers. The desired structure is chosen and the pertinent controller tags (typically at a minimum SP, PV, OP and MODE) are mapped into the tree. For this example, we will create a structure with two manufacturing sites and a small number of controllers at each site. In most industrial applications, the number of controllers would be much higher, and additional asset levels (departments, units, equipment, etc.) could of course be included. We use SPy.trees functionality within the Data Lab notebook to create the basic structure: Controller tags for SP, PV, OP, and MODE are identified using SPy.search. Cleansed controller names are extracted and inserted as asset names within the tree: Next, the controller tags (SP, PV, OP, and MODE), identified in the previous Data Lab code section using SPy.search, are mapped into the tree. At the same time, the second key step in creating the CLPM solution, developing basic performance metrics and calculations using Seeq Formula and inserting them into the asset tree, is completed. Note that in our example formulas, manual mode is detected when the numerical mode signal equals 0; this formula logic will need to be adjusted based on your mode signal conventions. While this second key step could be done just as easily as a separate code section later, it also works nicely to combine it with the mapping of the controller tag signals: STEP 2: Developing performance metric calculations and inserting them in the tree as formulas Several key points need to be mentioned related to this step in the CLPM process, which was implemented using the Data Lab code section above (see previous screenshot). There are of course many possible performance metrics of varying complexity. A good approach is to start with basic metrics that are easily understood, and to incrementally layer on more complex ones if needed, as the CLPM solution is used and shortcomings are identified. The selection of metrics, parameters, and the extent of customization for individual controllers should be determined by those who understand the process operation, control strategy, process dynamics, and overall CLPM objectives. The asset structure and functionality provided with the Data Lab asset tree creation enables the user to implement the various calculation details that will work best for their objectives. Above, we implemented Hourly Average Absolute Controller Error (as a percentage based on the PV value) and Hourly Percent Span Traveled as basic metrics for quantifying performance. When performance is poor (high variation), the average absolute controller error and percent span traveled will be abnormally high. Large percent span traveled values also lead to increased control valve maintenance costs. We chose to calculate these metrics on an hourly basis, but calculating more or less frequently is easily achieved, by substituting different recurring time period functions in place of the “hours()” function in the formulas for Hourly Average Absolute Controller Error and Hourly Percent Span Traveled. The performance metric calculations are inserted as formula items in the asset tree. This is an important aspect as it allows calculation parameters to be customized as needed on an individual controller basis, using Data Lab code, to give more accurate performance metrics. There are many customization possibilities, for example controller specific PV ranges could be used to normalize the controller error, or loosely tuned level controllers intended to minimize OP movement could be assigned a value of 0 error when the PV is within a specified range of SP. The Hourly Average Absolute Controller Error and Hourly Percent Span Traveled are then aggregated into an Hourly Loop Health Score using a simple weighting calculation to give a single numerical value (0-100) for categorizing overall performance. Higher values represent better performance. Another possible approach is to calculate loop health relative to historical variability indices for time periods of good performance specified by the user. The magnitude of a loop health score comprised of multiple, generalized metrics is never going to generate a perfect measure of performance. As an alternative to using just the score value to flag issues, the loop health score can be monitored for significant decreasing trends to detect performance degradation and report controller issues. While not part of the loop health score, note in the screenshot above that we create an Hourly Percent Time Manual Mode signal and an associated Excessive Time in Manual Mode condition, as another way to flag performance issues – where operators routinely intervene and adjust the controller OP manually to keep the process operating in desired ranges. Manual mode treemap visualizations can then be easily created for all site controllers. With the asset tree signal mappings and performance metric calculations inserted, the tree is pushed to a Workbench Analysis and the push results are checked: STEP 3: Calculating advanced metrics via scheduled Data Lab notebooks (if needed) Basic performance metrics (using Seeq Formula) may be all that are needed to generate actionable CLPM, and there are advantages to keeping the calculations simple. If more advanced performance metrics are needed, scheduled Data Lab notebooks are a good approach to do the required math calculations, push the results as signals into Seeq, and then map/insert the advanced metrics as items in the existing asset tree. There are many possibilities for advanced metrics (oscillation index, valve stiction, non-linearity measures, etc.), but here as an example, we calculate an oscillation index and associated oscillation period using the asset tree Controller Error signal as input data. The oscillation index is calculated based on the zero crossings of the autocorrelation function. Note: the code below does not account for time periods when the controller is in manual, the process isn’t running, problems with calculating the oscillation index across potential data gaps, etc. – these issues would need to be considered for this and any advanced metric calculation. Initially, the code above would be executed to fill in historical data oscillation metric results for as far back in time as the user chooses, by adjusting the calculation range parameters. Going forward, this code would be run in a recurring fashion as a scheduled notebook, to calculate oscillation metrics as time moves forward and new data becomes available. The final dataframe result from the code above looks as follows: After examining the results above for validity, we push the results as new signals into Seeq Workbench with tag names corresponding to the column names above. Note the new, pushed signals aren’t yet part of the asset tree: There are two additional sections of code that need to be executed after oscillation tag results have been pushed for the first time, and when new controller tags are added into the tree. These code sections update the oscillation tag metadata, adding units of measure and descriptions, and most importantly, map the newly created oscillation tags into the existing CLPM asset tree: STEP 4: Configuring asset scaled and individual controller visualizations in Seeq Workbench The CLPM asset tree is now in place in the Workbench Analysis, with drilldown functionality from the “US Sites” level to the signals, formula-based performance metrics, and Data Lab calculated advanced metrics, all common to each controller: The user can now use the tree to efficiently create monitoring and reporting visualizations in Seeq Workbench. Perhaps they start by setting up raw signal and performance metric trends for an individual controller. Here, performance degradation due to the onset of an oscillation in a level controller is clearly seen by a decrease in the loop health score and an oscillation index rising well above 1: There are of course many insights to be gained by asset scaling loop health scores and excessive time in manual mode across all controllers at a site. Next, the user creates a sorted, simple table for average loop health, showing that 7LC155 is the worst performing controller at the Beech Island site over the time range: The user then flags excessive time in manual mode for controller 2FC117 by creating a Lake Charles treemap based on the Excessive Time in Manual Mode condition: A variety of other visualizations can also be created, including controller data for recent runs versus current in chain view, oscillation index and oscillation period tables, a table of derived control loop statistics (see screenshot below for Lake Charles controller 2FC117) that can be manually created within Workbench or added later as items within the asset tree, and many others. Inspecting a trend in Workbench (see screenshot below) for a controller with significant time in manual mode, we of course see Excessive Time in Manual Mode capsules, created whenever the Hourly Percent Time Manual Mode was > 25% for at least 4 hours in a row. More importantly, we can see the effectiveness of including hours().intersect($Mode!=0) in the formulas for Hourly Average Absolute Controller Error and Hourly Percent Span Traveled. When the controller is in manual mode, that data is excluded from the metric calculations, resulting in the gaps shown in the trend. Controller error and OP travel have little meaning when the controller is in manual, so excluding data is necessary to keep the metrics accurate. It would also be very easy to modify the formulas to only calculate metrics for hourly time periods where the controller was in auto or cascade for the entire hour (using Composite Condition and finding hourly capsules that are “outside” manual mode capsules). The ability to accurately contextualize the metric calculations, to the time periods where they can be validly calculated, is a key feature in Seeq for doing effective CLPM implementations. Please also refer to the “Reminders and Other Considerations” section below for more advanced ideas on how to identify time periods for metric calculations. STEP 5: Setting up site performance as well as individual controller monitoring in Seeq Organizer As the final key step, the visualizations created in Workbench are inserted into Seeq Organizer to create a cohesive, auto-updating CLPM report with site overviews as well as individual controller visuals. With auto-updating date ranges applied, the CLPM reports can be “review ready” for recurring meetings. Asset selection functionality enables investigative workflows: poorly performing controllers are easily identified using the “Site CLPM” worksheet, and then the operating data and metrics for those specific controllers can be quickly investigated via asset selection in the site’s “Individual Controllers” worksheet – further details are described below. An example “Site CLPM” Organizer worksheet (see screenshot below) begins with a loop health performance ranking for each site, highlighting the worst performing controllers at the top of the table and therefore enabling the manufacturing team to focus attention where needed; if monitoring hundreds of controllers, the team could filter the table to the top 10 or 20 worst performing controllers. The visualizations also include a treemap for controllers that are often switched to manual mode – the team can talk to operators on each crew to determine why the controllers are not trusted in auto or cascade mode, and then generate action items to resolve. Finally, oscillating controllers are flagged in red in the sorted Oscillation Metrics tables, with the oscillation period values also sorted – short oscillation periods may prematurely wear out equipment and valves due to high frequency cycling; long oscillation periods are more likely to negatively impact product quality, production rate, and energy consumption. Controllers often oscillate due to root causes such as tuning and valve stiction and this variability can often be eliminated once an oscillating controller has been identified. The oscillation period table can also be perused for controllers with similar periods, which may be evidence of an oscillation common to multiple controllers which is generating widespread process variation. An example “Individual Controllers” Organizer worksheet is shown below, where detailed operating trends and performance metrics can be viewed for changes, patterns, etc., and chain view can be used to compare controller behavior for the current production run versus recent production runs. Other controllers can be quickly investigated using the asset selection dropdown, and the heading labels (e.g., Beech Island >> 7LC155) change dynamically depending on which controller asset is selected. For example, the Beech Island 7LC155 controller which was identified as the worst performing controller in the “Site CLPM” view above, can be quickly investigated in the view below, where it is evident that the controller is oscillating regularly and the problem has been ongoing, as shown by the comparison of the current production run to the previous two runs: Reminders and Other Considerations As evident with the key steps outlined above, a basic CLPM solution can be rapidly implemented in Seeq. While Seeq’s asset and contextualization features are ideal for efficiently creating CLPM, there are many details which go into developing and maintaining actionable CLPM dashboards for your process operation. A list of reminders and considerations is given below. 1. For accurate and actionable results, it is vital to only calculate performance metrics when it makes sense to do so, which typically means when the process is running at or near normal production rates. For example, a controller in manual during startup may be expected and part of the normal procedure. And of course, calculating average absolute controller error when the process is in an unusual state will likely lead to false indications of poor performance. Seeq is designed to enable finding those very specific time periods when the calculations should be performed. In the CLPM approach outlined in this article, we used time periods when the controller was not in manual by including hours().intersect($Mode!=0) in the formulas for Hourly Average Absolute Controller Error and Hourly Percent Span Traveled. When the controller is in manual mode, that data is excluded from the metric calculations. But of course, a controller might be in auto or cascade mode when the process is down, and there could be many other scenarios where only testing for manual mode isn’t enough. In the CLPM approach outlined above, we intentionally kept things simple by just calculating metrics when the controller was not in manual mode, but for real CLPM implementations you will need to use a more advanced method. Here are a few examples of finding “process running” related time periods using simple as well as more advanced ways. Similar approaches can be used with your process signals, in combination with value searches on controller mode, for excluding data from CLPM calculations: A simple Process Running condition can be created with a Value Search for when Process Feed Flow is > 1 for at least 2 hours. A 12 Hours after Process Running condition can be created with a formula based on the Process Running Condition: $ProcessRunning.afterStart(12hr) A Process Running (> 12 hours after first running) condition can then be created from the first two conditions with the formula: $ProcessRunning.subtract($TwelveHrsAfterRunning) Identifying time periods based on the value and variation of the production rate is also a possibility as shown in this Formula: The conditions described above are shown in the trends below: 2. As previously mentioned, including metric formulas and calculations as items within the asset tree enables customization for individual controllers as needed, when controllers need unique weightings, or when unique values such as PV ranges are part of the calculations. It may also be beneficial to create a “DoCalcsFlag” signal (0 or 1 value) as an item under each controller asset and use that as the criteria to exclude data from metric calculations. This would allow customization of “process is running normally and controller is not in manual” on an individual controller basis, with the result common for each controller represented as the “DoCalcsFlag” value. 3. In the CLPM approach outlined above, we used SPy.trees in Data Lab to create the asset tree. This is the most efficient method for creating large scale CLPM. You can also efficiently create the basic tree (containing the raw signal mappings) from a CSV file. For small trees (<= 20 controllers), it is feasible to interactively create the CLPM asset tree (including basic metric calculation formulas) directly in Workbench using Asset Groups. The Asset Groups approach requires no Python coding and can be quite useful for a CLPM proof of concept, perhaps focused on a single unit at the manufacturing site. For more details on Asset Groups: https://www.seeq.org/index.php?/forums/topic/1333-asset-groups-101-part-1/). 4. In our basic CLPM approach, we scoped the CLPM asset tree and calculations to a single Workbench Analysis. This is often the best way to start for testing, creating a proof of concept, getting feedback from users, etc. You can always decide later to make the CLPM tree and calculations global, using the SPy.push option for workbook=None. 5. For long-term maintenance of the CLPM tree, you may want to consider developing an add-on for adding new controllers into the tree, or for removing deleted controllers from the tree. The add-on interface could also prompt the user for any needed customization parameters (e.g., PV ranges, health score weightings) and could use SPy.trees insert and remove functionality for modifying the tree elements. 6. When evidence of more widespread variation is found (more than just variability in a single controller), and the root cause is not easily identified, CLPM findings can be used to generate a list of controllers (and perhaps measured process variables in close proximity) that are then fed as a dataset to Seeq’s Process Health Solution for advanced diagnostics. 7. For complex performance or diagnostic metrics (for example, stiction detection using PV and OP patterns), high quality data may be needed to generate accurate results. Therefore, some metric calculations may not be feasible depending on the sample frequency and compression levels inherent with archived and compressed historian data. The only viable options might be using raw data read directly from the distributed control system (DCS), or specifying high frequency scan rates and turning off compression for controller tags such as SP, PV, and OP in the historian. Another issue to be aware of is that some metric calculations will require evenly spaced data and therefore need interpolation (resampling) of historian data. Resampling should be carefully considered as it can be problematic in terms of result accuracy, depending on the nature of the calculation and the signal variability. 8. The purpose of this article was to show how to set up basic CLPM in Seeq but note that many types of process calculations to monitor “asset health” metrics could be created using a similar framework. Summary While there are of course many details and customizations to consider for generating actionable controller performance metrics for your manufacturing site, the basic CLPM approach above illustrates a general framework for getting started with controller performance monitoring in Seeq. The outlined approach is also widely applicable for health monitoring of other types of assets. Asset groups/trees are key to scaling performance calculations across all controllers, and Seeq Data Lab can be used as needed for implementing more complex metrics such as oscillation index, stiction detection, and others. Finally, Seeq Workbench tools and add-on applications like Seeq’s Process Health solution can be used for diagnosing the root cause of performance issues automatically identified via CLPM monitoring. CLPM Asset Tree Creation.ipynb Advanced CLPM Metric Calculations.ipynb
    6 points
  2. Tulip is one of the leading frontline operations platforms, providing manufacturers with a holistic view of quality, process cycle times, OEE, and more. The Tulip platform provides the ability to create user-friendly apps and dashboards to improve the productivity of your operations without writing any code. Integrating Tulip and Seeq allows Tulip app and dashboard developers to directly include best-in-class time series analytics into their displays. Additionally, Seeq can access a wealth of contextual information through Tulip Tables. Accessing Tulip Table Data in Seeq Tulip table data is an excellent source of contextual information as it often includes information not gathered by other systems. In our example, we will be using a Tulip Table called (Log) Station Activity History. This data allows us to see how long a line process has been running, the number of components targeted for assembly, actually assembled, and the number of defects. The easiest way to bring this into Seeq is as condition data. We will create one condition per station and each column will be a capsule property. This can be achieved with a scheduled notebook: import requests import json import pandas as pd # This method gets data from a tulip table and formats the data frame into a Seeq-friendly structure def get_data_from_tulip(table_id, debug): url = f"https://{TENANT_NAME}.tulip.co/api/v3/tables/{table_id}/records" headers = { "Authorization": AUTH_TOKEN } params = { "limit": 100, "sortOptions" : '[{"sortBy": "_createdAt", "sortDir": "asc"}]' } all_data = [] data = None while True: # Use for paginating the reqeusts if data: last_sequence = data[-1]['_sequenceNumber'] params['filters'] = json.dumps([{"field":"_sequenceNumber","functionType":"greaterThan","arg":last_sequence}]) # Make the API request response = requests.get(url, headers=headers, params=params) if debug: print(json.dumps(response.json(), indent=4)) # Check if the request was successful if response.status_code == 200: # Parse the JSON response data = response.json() all_data.extend(data) if len(data) < 100: break # Exit the loop if condition is met else: print(f"API request failed with status code: {response.status_code}") break # Convert JSON data to pandas DataFrame df = pd.DataFrame(all_data) df = df.rename(columns={'id': '_id'}) df.columns = df.columns.str.split('_').str[1] df = df.drop(columns=['sequenceNumber','hour'], errors='ignore') df['createdAt'] = pd.to_datetime(df['createdAt']) df['updatedAt'] = pd.to_datetime(df['updatedAt']) df = df.rename(columns={'createdAt': 'Capsule Start', 'updatedAt': 'Capsule End', 'duration': 'EventDuration'}) df = df.dropna() return df def create_metadata(station_data, station_name): print(f"DataFrame for station: {station}") print("Number of rows:", len(group)) metadata=pd.DataFrame([{ 'Name': station_name, 'Type': 'Condition', 'Maximum Duration': '1d', 'Capsule Property Units': {'status': 'string', 'id': 'string', 'station':'string', 'duration':'s'} }]) return metadata # This method splits the dataframe by station. Each Station will represent a condition in Seeq. def create_dataframe_per_station(all_data, debug): data_by_station = all_data.groupby('station') if debug: for station, group in data_by_station: print(f"DataFrame for station: {station}") print("Number of rows:", len(group)) display(group) return data_by_station # This method sends the data to Seeq def send_to_seeq(data, metadata, workbook, quiet): spy.push(data=data, metadata=metadata, workbook=workbook, datasource="Tulip Operations", quiet=quiet) data = get_data_from_tulip(TABLE_NAME, False) per_station = create_dataframe_per_station(data, False) for station, group in per_station: metadata = create_metadata(group, station) send_to_seeq(group, metadata, 'Tulip Integration >> Bearing Failure', False) The above notebook can be run on a schedule with the following command: spy.jobs.schedule('every 6 hours') This will pull the data from the Tulip Table into Seeq to allow for quick analysis. The notebook above will need you to provide a tenant, API key, and table name. It will also be using this REST API method to get the records. Once provided, this data will be pulled into a dataset called Tulip Operations and scoped to a workbook called Tulip Integration. We can now leverage the capsule properties to start isolating interesting periods of time. For example, using the formula $ea.keep('status', isEqualTo('RUNNING')) Where $ea is the Endbell Assembly condition from the Tulip Table. We can create a new condition keeping only the capsules where the state is running. Once a full analysis is created, Seeq content can be displayed in a Tulip App as an iFrame, allowing for the combination of Tulip and Seeq data: Data can be pushed back to Tulip using the create record API. This allows for Tulip Dashboards to contain Seeq Content:
    5 points
  3. A typical data cleansing workflow is to exclude equipment downtime data from calculations. This is easily done using the .remove() and .within() functions in Seeq formula. These functions remove or retain data when capsules are present in the condition that the user supplies as a parameter to the function. There is a distinct difference in the behavior of the .remove() and .within() functions that users should know about, so that they can use the best approach for their use case. .remove() removes the data during the capsules in the input parameter condition. For step or linearly interpolated signals, interpolation will occur across those data gaps that are of shorter duration than the signal's maximum interpolation. (See Interpolation for more details concerning maximum interpolation.) .within() produces data gaps between the input parameter capsules. No interpolation will occur across data gaps (no matter what the maximum interpolation value is). Let's show this behavior with an example (see the first screenshot below, Data Cleansed Signal Trends), where an Equipment Down condition is identified with a simple Value Search for when Equipment Feedrate is < 500 lb/min. We then generate cleansed Feedrate signals which will only have data when the equipment is running. We do this 2 ways to show the different behaviors of the .remove() and .within() functions. $Feedrate.remove($EquipmentDown) interpolates across the downtime gaps because the gap durations are all less than the 40 hour max interpolation setting. $Feedrate.within($EquipmentDown.inverse()) does NOT interpolate across the downtime gaps. In the majority of cases, this result is more in line with what the user expects. As shown below, there is a noticeable visual difference in the trend results. Gaps are present in the green signal produced using the .within() function, wherever there is an Equipment Down capsule. A more significant difference is that depending on the nature of the data, the statistical calculation results for time weighted values like averages and standard deviations, can be very different. This is shown in the simple table (Signal Averages over the 4 Hour Time Period, second screenshot below). The effect of time weighting the very low, interpolated values across the Equipment Down capsules when averaging the Feedrate.remove($EquipmentDown) signal, gives a much lower average value compared to that for $Feedrate.within($EquipmentDown.inverse()) (1445 versus 1907). Data Cleansed Signal Trends Signal Averages over the 4 Hour Time Period Content Verified DEC2023
    4 points
  4. Hi Jacob. you can use the within() function to include only the portions of the signal while the condition appears and then use this signal in your calculations: Regards, Thorsten
    3 points
  5. Here's an alternative method to getting the last X batches in the last 30 days: // return X most recent batches in the past Y amount of time $numBatches = 20 // X $lookback = 1mo // Y // create a rolling condition with capsules that contain X adjacent capsules $rollingBatches = $batchCondition.removeLongerThan($lookback) .toCapsulesByCount($numBatches, $lookback) // find the last capsule in the rolling condition that's within the lookback period $currentLookback = capsule(now()-$lookback, now()) $batchWindow = condition( $lookback, $rollingBatches.toGroup($currentLookback, CAPSULEBOUNDARY.ENDSIN).last() ) // find all the batches within the capsule identified // ensure all the batches are within the lookback period $batchCondition.inside($batchWindow) .touches(condition($lookback, $currentLookback)) This is similar to yours in that it uses toGroup, but the key is in the use of toCapsulesByCount as a way to get a grouping of X capsules in a condition. You can see an example output below. All capsules will show up as hollow because by the nature of the rolling 'Last X days' the result will always be uncertain.
    3 points
  6. Hi Sam, Thanks for your response and the additional information. To create the table you need, we can utilize the transform function along with a capsule expression to iterate through each product run capsule. Within this loop we will calculate the previous year averages and dynamically set these to capsule properties based on the Product ID. If you're not familiar with transforms and capsule expressions, here is a good Tips and Tricks article. Below is an example of formula code you can integrate with your existing Product Type signal to condition formula. // existing $product_type.toCondition('Product ID') code here ... // I am assuming this is defined as $product_condition // if necessary, define your measurement signals filtered for $good_runs $a_good = $a.within($good_runs) $b_good = $b.within($good_runs) $b_good = $b.within($good_runs) $months = months() // monthly periodic condition // Loop through each capsule in $product_condition and calculate last year's measurement averages $product_condition.transform($capsule -> { $productid = $capsule.property('Product ID') // get Product ID from the current capsule // Extract a monthly capsule from $months one year prior to the current capsule. Note that your current capsule could span two months. // .first() selects the month based on the start month of the capsule. Change this to .last() if you want the ending month. $previous_year_month_capsule = $months.toGroup(capsule($capsule.startKey()-1y,$capsule.endKey()-1y)).first() // Calculate scalar averages over the previous year month capsule, filtered for current product ID $a_avg = $a_good.within($product_condition.keep('Product ID', isEqualTo($productid))).average($previous_year_month_capsule) $b_avg = $b_good.within($product_condition.keep('Product ID', isEqualTo($productid))).average($previous_year_month_capsule) $c_avg = $c_good.within($product_condition.keep('Product ID', isEqualTo($productid))).average($previous_year_month_capsule) // Set averages as capsule properties $capsule.setProperty('Previous Year Month Average A', $a_avg) .setProperty('Previous Year Month Average B', $b_avg) .setProperty('Previous Year Month Average C', $c_avg) }) From here, create a Condition Table and use the Columns selector to pick each of the Capsule properties you want in the table. Here's an example of how you might set up the Condition Table using the Example >> Batch Data >> Reactor 1 data source: We'll mostly ignore my previous reply 😅 - helpful if you want the previous year average signals for analysis but not for the table filtered by Product ID. Hope this helps! Dan
    2 points
  7. There are instances where it's desirable to add or remove Access Control to a user created datasource. For instance, you may want to push data from Seeq Datalab to a Datasource that is only available to a sub-set of users. Below is a tip how to add additional entries to a Datasource Access Control List (ACL) using `spy.acl`. 1) Identify Datasource ID If not already known, retrieve the ID for the Datasource. This can be done via the Seeq REST API (accessible via the "hamburger" menu in the upper right corner of your browser window). This is different from the datasourceId identifier and will contain a series of characters as shown below. It can be retrieved via the `GET/datasources` endpoint under the `Datasources` section. The response body will contain the ID: 2) Identify User and/or Group ID We will also want to identify the User and/or Group ID. Below is an example of how to retrieve a Group ID via the `UserGroups` GET/usergroups endpoint: The response body will contain the ID: 3) Use `spy.acl()` to add group to the Datasource Access Control List (ACL) Launch a Seeq Datalab Project and create a new ipynb Notebook Assign the two IDs to a variable datasource_id='65BDFD4F-1FA3-4EB5-B21E-069FA5A772DF' group_id='19F1B7DC-69BE-4402-AD3B-DF89B1B9A1A4' (Optional) - Check the current Access Control List for the Datasource using `spy.acl.pull()` current_access_control_df=spy.acl.pull(datasource_id) current_access_control_df.iloc[0]['Access Control'] Add the group_id to the Datasource ACL using `spy.acl.push()` spy.acl.push(datasource_id, { 'ID': group_id, 'Read': True, 'Write': True, 'Manage': False }, replace=False) Note I've set `replace=False`, which means the group will be appended to the current access list. If the desire is to the replace the entire existing ACL list, this can be toggled to `replace=True`. Similarly, you can adjust the read/write/manage permissions to fit your need. For more information on `spy.acl.push`, reference the `spy.acl.ipynb` document located in the SPy Documentation folder or access the docstring by executing help(spy.acl.push) (Optional) - To confirm the correct ID has been added, re-pull the ACL via `spy.acl.pull()` current_access_control_df=spy.acl.pull(datasource_id) current_access_control_df.iloc[0]['Access Control']
    2 points
  8. Starting in R63, you have the ability to "deep copy" worksheets to automatically create new, independent items. Details here: https://support.seeq.com/kb/latest/cloud/worksheet-document-organization
    2 points
  9. Working with Asset Groups or spy.assets.Tree() has been effective. However, as your project expands, the process of adding and managing additional assets has become cumbersome. It's time to elevate your approach by transitioning to an Asset Template for building your Asset Tree. In this guide, I'll share valuable tips and tricks for creating an asset tree from an existing hierarchy. If you already have an established asset tree from another source, such as PI AF, you can leverage it as a quick starting point to build your tree in Seeq. Whether you're starting with an existing asset tree or starting from scratch, these insights will help you seamlessly integrate your asset structure into Seeq Data Lab. Additionally, I'll cover essential troubleshooting techniques, focusing on maximizing ease of use when using Asset Templates. In a future topic, I'll discuss the process of building an Asset Tree from scratch using Asset Templates. Let's dive into the best practices for optimizing your asset management workflow with Seeq Data Lab. Why Templates over Trees? The choice between using Asset Templates and spy.assets.Tree() depends on the complexity of your project and the level of customization you need. Here are some reasons why you might want to use Asset Templates instead of spy.assets.Tree(): Object-Oriented Design: Asset Templates allow you to leverage object-oriented design principles like encapsulation, inheritance, composition, and mixins. This can increase code reuse and consistency, especially in larger projects. However, the learning curve is a bit longer if you do not come from a programming background. Customization: Asset Templates provide a higher level of customization. You can define your own classes and methods, which can be useful if you need to implement complex logic or calculations. You can more easily create complex hierarchical trees similar Asset Trees from PI AF. Handling Exceptions: Manufacturing scenarios often have many exceptions. Asset Templates can accommodate these exceptions more easily than spy.assets.Tree(). Here are some reasons why you might want to use spy.assets.Tree() instead of Asset Templates: Simplicity: spy.assets.Tree() is simpler to use and understand, especially for beginners or for simple projects. If you just need to create a basic hierarchical structure without any custom attributes or methods, spy.assets.Tree() is a good choice. You can always graduate to Asset Templates later. Flat Structure: If your data is already organized in a flat structure (like a CSV file), spy.assets.Tree() can easily convert that structure into an asset tree. Less Code: With spy.assets.Tree(), you can create an asset tree with just a few lines of code. You don't need to define any classes or methods. Okay, we are settled on Asset Templates. What's first? If you have an existing asset hierarchy with the attributes you want to use in your calculations, you can start there. I'll be using the Example >> Cooling Tower 1 as my example existing Asset Tree. We'll use the asset tree path for our spy.search criteria and use the existing asset hierarchy when building our Template. The attachment includes a complete working Jupyter notebook. You can upload it to a Data Lab project and test each step as described here. Okay, let's review the code section by section. # 1. Importing Libraries from seeq import spy import pandas as pd import re from seeq.spy.assets import Asset # 2. Checking and displaying the spy version version = spy.__version__ print(version) # 3. Setting the compatibility option so that you maximize the chance that SPy # will remain compatible with your notebook/script spy.options.compatibility = 189 # 4. Disply configuration for Pandas to show all columns in DataFrame output pd.set_option('display.max_colwidth', None) In this cell, the script initializes the Seeq Python (SPy) library as well as other required libraries, checks the spy version, sets a compatibility option, and configures pandas to display DataFrame output without column width truncation. # 1. Use spy.search to search for assets using the existing Asset Tree hierarchy. metadata = spy.search( {'Path': 'Example >> Cooling Tower 1 >> Area*'}, # MODIFY: Change the search path old_asset_format=False, limit=None, ) # 2. Split the "Path" column by the ">>" delimiter and create a new column "Modified_Path." metadata["Modified_Path"] = metadata['Path'] + ' >> ' + metadata['Asset'] split_columns = metadata['Modified_Path'].str.split(' >> ', expand=True) # 3. Rename the columns of the split DataFrame to represent different hierarchy levels. split_columns.columns = [f'Level_{i+1}' for i in range(split_columns.shape[1])] # 4. Concatenate the split columns with the original DataFrame to incorporate the hierarchy levels. metadata = pd.concat([metadata, split_columns], axis=1) # 5. Required for building an asset tree. This is the parent branch/name of the asset tree. metadata['Build Path'] = 'Asset Tree Descriptive Name' # MODIFY: Set the desired parent branch name # 6. The child branch of 'Build Path' and will be named after the "Level_2" column. metadata['Build Asset'] = metadata['Level_2'] # Ex: "Cooling Tower 1" This cell constructs the metadata used in our new asset tree. The spy.search on the existing hierarchy is used to retrieve the metadata for each attribute in the asset path. The existing tree path is "Example >> Cooling Tower 1 >> Area*". This will fetch all attributes under each of the Areas (e.g., Area A - K). However, it will not retrieve any attributes that may reside in "Example >> Cooling Tower 1. The complete path for each asset is the concatenation of the "Path" column and the "Asset" column. This "Modified Path" is how we will define each level of our new Asset Tree. We are splitting the Modified Path and inserting each level as a new column that will be referenced later when building the asset tree. In steps 5 and 6, we define the first two levels of the asset tree. In this example, the new path will start with "Asset Tree Descriptive Name >> Cooling Tower 1". By utilizing the 'Level_2' column in the 'Build Path,' if we had included all cooling towers in our spy.search, the new second level in the path could be 'Cooling Tower 2,' depending on the asset's location in the existing tree." Alternatively, if you prefer a flat tree, you can specify the name of "Level 2" as a string rather than referencing the metadata column. # This is the class will be called when using Spy.Asset.Build # This is the child branch of 'Build Asset' # Ex: Asset Tree Descriptive Name >> Cooling Tower 1 >> Area A # Can you also have attributes at these levels by adding @Asset.Attribute() within the class. class AssetTreeName(Asset): # MODIFY: AssetTreeName @Asset.Component() def Asset_Component(self, metadata): # MODIFY: Unit_Component return self.build_components( template=Level_4, # This the name of the child branch class metadata=metadata, column_name='Level_3' # Metadata column name of the desired asset; Ex: Area A ) # Example Roll Up Calculation # This roll up calculation will find the evaluate the temperature if each child branch (e.g., Area A, Area B, # Area C, etc.) and create a new signal of the maximum temperature of the group @Asset.Attribute() def Cooling_Tower_Max_Temperature(self, metadata): return self.Asset_Component().pick({ ## MODIFY: Unit_Component() if changed in @Asset.Component() above 'Name': 'Temperature Unique Raw Tag' # MODIFY:'Temperature Unique Raw Tag'; this is the Friendly name of the attribute created in Level_4 }).roll_up('maximum') # MODIFY: 'maximum' In this cell, we are building our first class in our Asset Template. In the context of Asset Templates, consider a class as a general contractor responsible for constructing the current level in your asset tree. It subcontracts the responsibility for the lower level to that level's class. TIn the class, you define attributes such as signals, calculated signals, conditions, scorecard metrics, etc. or components, which are the 'child' branches of this 'parent' branch. These components are built by a different class. This first class will be called when we use spy.assets.build to create the new asset tree. @Asset.Component() indicates that we are defining a method to construct a component. The method name is Asset_Component which will construct the child branch. The parameter 'template' defines the class that will construct the child branch, while 'column_name' specifies the name of each child branch. Here we are using "Level_3" which contains the Area name (e.g., Area A). @Asset.Attribute() indicates that we are defining a method that will be an attribute. More details about creating attributes will be discussed in the next class. However, the example provided is a roll up calculation which will be a signal equal to the maximum value of each of the child branches' "Temperature Unique Raw Tag" which will be defined in the next class. When using roll-up calculations, note that the method being called is the component method responsible for constructing the child branches, not the attribute residing in the child branches. The .pick() function utilizes the friendly name of the attribute to identify the attributes in the child branches. # This is the child branch of 'AssetTreeName' class class Level_4(Asset): @Asset.Attribute() # Raw Tag (Friendly Name is the same as the method name with '_' removed) # Use this method if you are highly confident there are no multiple matches in current or child branches def Temperature_Unique_Raw_Tag(self, metadata): # MODIFY: Temperature_Unique_Raw_Tag return metadata[metadata['Name'] == 'Temperature'] @Asset.Attribute() def Temperature_Possible_Duplicates_Raw_Tag(self, metadata): # MODIFY:Temperature_Possible_Duplicates_Raw_Tag # If multiple matching metadata is found, the first match is selected using .iloc[[0]]. # This also addresses duplicate attributes in child branches. # If selecting first match is not desired, improve regex search criteria # or add additional filters to select desired attribute (e.g, # metadata[metadata['Name'].str.contains(r'(?:Temperature)$') # & metadata[metadata['Description'].str.contains(r'(?:Discharge)')] # ) filtered = metadata[metadata['Name'].str.contains(r'(?:Temperature)$')] # MODIFY: r'(?:Temperature)$') return filtered.iloc[[0]] In our example, Level_4 is the lowest level in our asset tree. All our attributes related to an asset will reside here. In our example, our raw tags (i.e., no calculations) will reside in the same branch as our derived tags (i.e., calculated), which typically reference the raw tags in the formula parameter. Each attribute is defined by a method starting with @Asset.Attribute before defining the method. When not specified, the friendly name of the attribute is the same as the method name with the underscores removed. Example: The method name is Temperature_Unique_Raw_Tag, and the friendly name is "Temperature Unique Raw Tag". We reference the metadata dataframe with a search criteria to select the proper row for the attribute. There are several methods using pandas to select the correct row. Here we are using the "Name" to find the row with "Temperature" as the value. Spy will find each "Temperature" value for each asset. This method can result in an error if there are multiple matches. The second method "Temperature_Possible_Duplicates_Raw_Tag" explains how you can avoid this error by using .iloc[[0]] to select the first match found or how you can increase the complexity of your search criteria to further filter your results. # Calculated Tag (Signal) referencing an attribute in this class @Asset.Attribute() def Temperature_Calc(self, metadata): return { 'Name': 'Signal Friendly Name', # MODIFY: 'Signal Friendly Name' 'Type': 'Signal', # This is the same formula you would use in Workbench Formula 'Formula': '$signal.agileFilter(1min)', # MODIFY: "$signal.agileFilter(1min)"; 'Formula Parameters': {'$signal': self.Temperature_Unique_Raw_Tag()}, # MODIFY: '$signal' and self.Temperature_Unique_Raw_Tag() } # Calculated Tag (Signal) not previously referenced as an attribute @Asset.Attribute() def Wet_Bulb(self, metadata): filtered = metadata[metadata['Name'].str.contains(r'^Wet Bulb$')] # MODIFY: '^Wet Bulb$' return { 'Name': 'Signal Friendly Name 2', # MODIFY: 'Signal Friendly Name 2' 'Type': 'Signal', 'Formula': '$a.agileFilter(1min)', # MODIFY: '$a.agileFilter(1min)' 'Formula Parameters': {'$a': filtered.iloc[[0]]}, # MODIFY: '$a' } # Calculated Tag (Condition) @Asset.Attribute() def Temperature_Cond(self, metadata): return { 'Name': 'Condition Friendly Name', # MODIFY: 'Condition Friendly Name' 'Type': 'Condition', 'Formula': '$b > 90', # MODIFY: '$b > 90' 'Formula Parameters': {'$b': self.Temperature_Possible_Duplicates_Raw_Tag()}, # MODIFY: '$b' & self.Temperature_Possible_Duplicates_Raw_Tag() } # Scorecard Metric # See Asset Templates documentation for more examples of Scorecard Metrics @Asset.Attribute() def Temperature_Statistic_KPI(self, metadata): return { 'Type': 'Metric', 'Measured Item': self.Temperature_Unique_Raw_Tag(), # MODIFY: self.Temperature_Unique_Raw_Tag() 'Statistic': 'Average' # MODIFY: 'Average' } This is a continuation of our Level_4 class. Here we are creating derived attributes which use a formula or metric and reference other attributes in most cases. There are several different example attributes for calculated signals, conditions, and scorecard metrics. You can specify the friendly name in the "Name" parameter as an alternative method to using the name of the method. The value of the "Formula" is the exact same formula you use in Workbench. "Formula Parameters" is where you define your variables in your formula. The value of the "variable" key (i.e., $signal) is a method call to the attribute to be used in the formula (i.e., self.Temperature_Unique_Raw_Tag()). If you wish to specify a signal that has not been defined as an attribute, you can specify the value of the referenced variable using the same search criteria when specifying a new attribute. It is important to remember to change the "Type" when creating a condition or a scorecard metric, as this will cause an error if not properly specified. # Use this method if you want a different friendly name rather than the regex search criteria @Asset.Attribute() def Friendly_Name_Example(self, metadata): return self.select_metadata(metadata, name_contains = r'^Compressor Power$', friendly_name ='Power') #MODIFY: name_contains = r'Compressor Power', friendly_name = 'Power' # Use this method if you want the friendly name to be the same as regex search criteria # without the regex formatting @Asset.Attribute() def No_Friendly_Name_Example(self, metadata): return self.select_metadata(metadata, r'^Compressor Power$', None) # Example of a string attribute being passed through @Asset.Attribute() def String_Example(self, metadata): return self.select_metadata(metadata, r'Compressor Stage$', None) # Example of an attribute not found @Asset.Attribute() def No_Attribute_Exists(self, metadata): return self.select_metadata(metadata, r'Non-existing Attribute', None) # 1. Method below matches the metadata, perform attribute type detection, and applies proper formula def select_metadata(self, metadata, name_contains, friendly_name): filtered = metadata[metadata['Name'].str.contains(name_contains)] if not filtered.empty: # checks if metadata is a signal vs scalar and selects the first signal type if filtered['Type'].str.contains('signal', case=False).any() : signal_check = filtered[filtered['Type'].str.contains('signal', case=False) & ~filtered['Value Unit Of Measure'].str.contains('string', case=False)] filtered_signal = signal_check if len(signal_check) > 0 else filtered.iloc[[0]] selected_metadata = filtered_signal.iloc[[0]] else: selected_metadata = filtered.iloc[[0]] return self.determine_attribute_type(selected_metadata, name_contains, friendly_name) else: # This returns a signal with no values rather than not adding an attribute when # an attribute cannot be found. If this is not desired, simply comment out the # else statement. Having a signal is recommended to assist in asset swapping. return { 'Name' : friendly_name if friendly_name is not None else re.sub( r'\^|\(|\$|\(\?:|\)$|\)', '', name_contains), 'Type': 'Signal', 'Formula': 'Scalar.Invalid.toSignal()', } # 2. Determine if an attribute is a signal or scalar type def determine_attribute_type(self, metadata, name_contains, friendly_name): if metadata['Type'].str.contains('signal', case=False).any(): return self.generic_metadata_function(metadata, name_contains, 'Signal', friendly_name) elif metadata['Type'].str.contains('scalar', case=False).any(): return self.generic_metadata_function(metadata, name_contains, 'Scalar', friendly_name) return None # 3. Creates the metadata to create the attribute def generic_metadata_function(self, metadata, name_contains, formula_type, friendly_name): if friendly_name is None: # Regular expression pattern to remove caret (^), first "(", dollar ($), # non-capturing groups (?:), and trailing ")" pattern = r'\^|\(|\$|\(\?:|\)$|\)' friendly_name = re.sub(pattern, '', name_contains) # Metadata for signals includes strings. Required to separate strings to perform a # formula on non-string signals. Check if the signal is a "string" based on 'Value Unit Of Measure' if formula_type == 'Signal' and metadata['Value Unit Of Measure'].str.contains( 'string', case=False).any(): formula_type = 'String' # If signal, perform a calculation. If string or scalar, pass only the variable in formula. if formula_type == 'Signal': formula = '$signal.validValues()' # MODIFY: '$signal.validValues()' elif formula_type == 'String': formula = '$signal' else: formula = '$scalar' return { 'Name' : friendly_name, 'Type': formula_type, 'Formula': formula, 'Formula Parameters': ( {'$signal': metadata} if formula_type in ['Signal', 'String'] else {'$scalar': metadata}) } This is a continuation of the Level_4 class. This section is more of a trick than a tip. A common request is how to bulk apply a formula to all signals as an initial data cleaning step. This can be challenging if your existing asset tree contains signals, scalars, and/or strings with the same friendly name in the parent and/or child branches. I've seen this happen when an instrument does not exist for a particular asset. The attribute still exists in the asset tree but contains no data and is either a scalar or string. This method allows users to quickly perform the same calculations on all raw data signals. It will check for string, scalar, and signal types to determine if a calculation is required. If there are multiple matches, it will preferentially select a signal over a string or scalar. It will perform the appropriate calculation on the tag based on the type or return a blank signal if the attribute is not found. The blank signal is a necessary evil for asset swapping if the attribute is used in a calculation for other attributes. The formula should be adjusted to handle when there are no valid values in the attribute. @Asset.Attribute() def Friendly_Name_Example(self, metadata): return self.select_metadata(metadata, name_contains = r'^Compressor Power$', friendly_name ='Power') This attribute method has been modified to make a method call to select_metadata. It passes the metadata, name_contains, which is the search criteria for finding the name value in the metadata "Name" column, and a friendly name. In this example, the attribute will be named "Power". @Asset.Attribute() def No_Friendly_Name_Example(self, metadata): return self.select_metadata(metadata, r'^Compressor Power$', None) If the friendly name does not need to be different from the name_contains parameter without the regex formatting (i.e., special characters), it will name the attribute the search criteria name minus special characters. In the example above the attribute will be named "Compressor Power". This naming method is fairly simple and is not dynamic enough to insert result of the regex search. In those cases, the friendly_name should be specified. The subsequent method call to determine_attribute_type determines the attribute type (e.g., signal, string, scalar). If multiple matches are found, it will select the first signal type over a scalar or string. If attribute is not found, it creates an empty attribute. The final method call to generic_metadata_function determines the friendly name to be used and selects the correct formula for the proper attribute type. As I said earlier, this is more of a trick that address most common issues I have faced when dealing with complex asset trees and applying a common formula in bulk. This can also be applied even if you don't want to apply a formula to all signals. In that case, you would simply modify the formula to be "$signal" instead of "$signal.validValues()". You can still preferentially select signals types over string or scalar types. However, if referencing one of these attributes in a calculated attribute later, you will still have to use the attribute method name (e.g., self.Friendly_Name_Example() when referencing the 'Power' attribute in the formula parameter). build_df = spy.assets.build(AssetTreeName,metadata) # Check for success for each tag for an asset build_df.head(15) After creating our asset structure and attributes, you can run this cell to build our asset tree. I recommend visually checking for success in the 'Build Result' column of the dataframe, looking for 'Success' in each attribute for a single asset. workbookid = "WORKBOOK_ID" # MODIFY: "WORKBOOK_ID" # If you want the tree to be available globally use None as the workbookid push_results_df = spy.push(metadata=build_df, workbook=workbookid, errors='catalog') # Check for success at Push Result push_results_df.head(15) Once you are satisfied with your results or have resolved any errors, you can push your asset tree. If you wish to push your tree globally, change the value of the 'workbook' argument in spy.push to be set to None. I recommend doing this only after addressing all the issues in a workbook, which can serve as your sandbox environment. errors = push_results_df[push_results_df['Push Result'] != "Success"] errors The final step I perform is to check if any of my attributes were not successfully pushed. If there were no issues, the error dataframe should contain no rows. Well, if you have made it to the end, congrats! If you skipped to the end, that's fine too. In this guide, we delved into the world of Asset Templates and their role in streamlining the creation and management of asset trees within Seeq Data Lab. By leveraging object-oriented design principles, encapsulation, and inheritance, Asset Templates offer a powerful way to build complex hierarchical structures. Whether you're dealing with an existing asset tree or starting from scratch, the insights shared here aim to maximize the efficiency of your workflow. Download the attached Jupyter notebook and experiment with the code by modifying or adding new attributes or applying it to a different asset tree. Remember, the best way to grasp these concepts is to dive into the provided Jupyter notebook, make modifications, and witness the results firsthand. In the comments below, I will add some screenshots of common errors I come across when using spy.assets.build and what you should do to troubleshoot the errors. Tips Tricks for Existing Asset Trees.ipynb
    2 points
  10. Troubleshooting common errors when using Asset Templates: Spy.Assets.Build Error 1: No matching metadata row found Error - Key Phrase: "No matching metadata row found" for Temperature Unique Raw Tag on Level_4 class Fix - Modify the search criteria of the attribute shown in the error Error 2: Component dependency not built (calculated tag) Errors - Key phrases: Component Dependency not built - A calculated signal could not be created because the formula parameter referenced was not created. Attribute dependency not built - No matching metadata row found for the referenced formula parameter. Fix - Modify the search criteria of the attributes referenced by the calculated signals Error 3: Component dependency not built (branch not built) Error: Key Phrases: name 'Level 4' is not defined. Unit Component [on AssetTreeName class] component dependency not built. This is because the referenced template in def Unit_Component in the AssetTreeClass does not exist. The class name for the child branch is actually "Not_Level_4". Fix- Change the template to match the actual class name. Example: Code causing error: class AssetTreeName(Asset): @Asset.Component() def Unit_Component(self, metadata): return self.build_components( template=Level_4, metadata=metadata, column_name='Level_3' ) class Not_Level_4(Asset): @Asset.Attribute() def Temperature_Unique_Raw_Tag(self, metadata): return metadata[metadata['Name'] == 'Temperature Unique Raw Tag'].iloc[[0]] Corrected Code: class Level_4(Asset): @Asset.Attribute() def Temperature_Unique_Raw_Tag(self, metadata): return metadata[metadata['Name'] == 'Temperature Unique Raw Tag'].iloc[[0]] Error 4: Multiple attributes returned Error - Key Phrase: "Multiple attributes returned" - This indicates that your search criteria for the attribute returned multiple results. In this case, there a match in ">> Area A" and a match in ">> Area A >> Area A-Duplicate". Fix - Use .iloc[[0]] at the end of the metadata search criteria to select the first match. Original: metadata[metadata['Name'].str.contains(r'(?:Temperature)')] Revised: metadata[metadata['Name'].str.contains(r'(?:Temperature)')].iloc[[0]] Alternatively, you can add more criteria to the search to filter down to the desired tag: metadata[metadata['Name'].str.contains(r'(?:Temperature)')] & metadata[metadata['Level_4'].str.contains(‘Duplicate’) Error 5: ‘class name’ object has no attribute ‘attribute method name’ for calculated attributes Error: Key Phrases: "in Temperature_Cond ‘Formula Parameters’ lies the error. For the formula parameter, we have misspelled the attribute method name in the Level_4 class. Fix – check for misspelled methods or correct method is referenced Error 6: ‘class name’ object has no attribute ‘attribute name’ for roll up calculations Error: Key Phrases: "Cooling_Tower_Max_Temperature" is causing the error. For the roll up calculation, we are trying to reference the self.Temperature_Unique_Raw_Tag() which is in Level_4 which is the child branch. Since this method does not reside in the AssetTreeName class, it cannot find it. Example: Code causing error: class AssetTreeName(Asset): @Asset.Component() def Unit_Component(self, metadata): return self.build_components( template=Level_4, # This the name of the child branch class metadata=metadata, column_name='Level_3' # Metadata column name of the desired asset; Ex: Area A ) # Example Roll Up Calculation @Asset.Attribute() def Cooling_Tower_Max_Temperature(self, metadata): return self.Temperature_Unique_Raw_Tag().roll_up('maximum') Fix - Use the method that builds the child branch (Unit_Component) in the AssetTreeName class and then use the pick function using the friendly name of the tag to be used in the calculation. # Example Roll Up Calculation @Asset.Attribute() def Cooling_Tower_Max_Temperature(self, metadata): return self.Unit_Component().pick({ 'Name': 'Temperature Possible Duplicates Raw Tag' }).roll_up('maximum') Error 7: Spy.Push Errors In this example, I have tried to apply an agileFilter function onto a string attribute. This error can occur when an asset may have a string or scalar attribute exist instead of a signal attribute when a field instrument does not exist for this asset while a sibling asset would have a signal attribute and you want to apply the agileFilter. It can be difficult to determine which attributes are causing the issue. You can catalog the errors and allow spy.push to continue to push the Asset Tree without the errors and cataloging the errors. push_results_df = spy.push(metadata=build_df, workbook=workbookid, errors='catalog') The error will be cataloged in the "Push Result" column with same error shown in the example. To view all errors, you can filter the push result to see where the push result was not success. error = push_results_df[push_results_df['Push Result'] != "Success"] error
    2 points
  11. I was able to use the template by using the 'copy' function as below. So grabbed the document template (as described in the article you've shared above), made a copy of it and assigned to a new document. new_document_name = 'Executive View: September' new_document = template_document.copy(label=new_document_name) new_document.name = new_document_name Then defined the parameters: new_document.parameters = { "month": 'September', "[Image]My Visualization 1": 'fig1.png', "[Image]My Visualization 2": 'fig2.png', "First App Impact": None, "First App": None } Appended it to my existing topic: topic.documents.append(new_document) And finally, pushed my topic back to seeq: spy.workbooks.push(topic)
    2 points
  12. Hi Johannes, maybe one quick way you could try: First you can move the signal by a longer period than the 90 minutes to make sure the value of the signal is present at the timestamp of the whole hour. Here I used $signal.move(-100min) Next you can resample the moved signal: $movedSignal.resample(1h) Regards, Thorsten
    2 points
  13. There are times when you may need to calculate a standard deviation across a time-range using the data within a number of signals. Consider the below example. When a calculation like this is meaningful/important, the straightforward options in Seeq may not be mathematically representative to calculate a comprehensive standard deviation. These straightforward options include: Take a daily standard deviation for each signal, then average these standard deviations Take a daily standard deviation for each signal, then take the standard deviation of the standard deviations Create a real-time standard deviation signal (using stddev($signal1, $signal2, ... , $signalN)), then take the daily average or standard deviation of this signal While straightforward options may be OK for many statistics (max of maxes, average of averages, sum of totalizes, etc), a time-weighted standard deviation across multiple signals presents an interesting challenge. This post will detail methods to achieve this type of calculation by time-warping the data from each signal then combining each individually warped signal into a single signal. Similar methods are also discussed in the following two seeq.org posts: Two different methods to arrive at the same outcome will be explored. Both of these methods share the same Step 1 & 2. Step 1: Gather Signals of Interest This example will consider 4 signals. The same methods can be used for more signals, but note that implementing this solution programmatically via Data Lab may be more efficient when considering a high number of signals (>20-30). Step 2: Create Important Scalar Constants and Condition Number of Signals: The number of signals to be considered. 4 in this case. Un-Warped Interval: The interval you are interested in calculating a standard deviation (I am interested in a Daily standard deviation, so I entered 1d) Warped Interval: A ratio calculation of Un-Warped Interval / Number of Signals. This metric is detailing what the new time-range will be for the time-warped signals. I.e. given I have 4 signals considering a days worth of data of, each signal's day worth of data will be warped into 6 hour intervals Un-Warped Periods: This creates a condition with capsules spanning the original periods of interest. periods($unwarped_interval) Method 1: Create ONE Time-Shift Signal, and move output Warped Signals The Time Shift Signal will be used as a counter to condense the data in the period of interest (1 day for this example) down to the warped interval (6 hours for this example). 0-timeSince($unwarped_period, 1s)*(1-1/$num_of_signals) The next step is to use this Time Shift Signal to move the data within each signal. Note there is an integer in this Formula that steps with each signal applied to. Details can be viewed in the screenshots. $area_a.move($time_shift_signal, $unwarped_interval).setMaxInterpolation($warped_interval).move(0*$warped_interval) The last step is to combine each of these warped signals together. We now have a Combined Output that can be used as an input into a Daily Standard Deviation that will represent the time-weighted standard deviation across all 4 signals within that day. Method 2: Create a Time-Shift Signal per each Signal - No Need to move output Warped Signals This method takes advantage of 4 time-shift signals, one per signal. Note there is also an integer in this Formula that steps with each signal applied to. Details can be viewed in the screenshot. These signals take care of the data placement, where-as the data placement was taken care of using .move(N*$warped_interval) above. 0*$warped_interval-timeSince($unwarped_period, 1s)*(1-1/$num_of_signals) We can then follow Method 1 to use the time shift signals to arrange our signals. We just need to be careful to use each time shift signal, as opposed to the single time shift signal that was created in Method 1. As mentioned above, there is no longer a .move(N*$warped_interval) needed at the end of this formula. The last step is to combine each of these warped signals together, similar to Method 1. $area_a.move($time_shift_1, $unwarped_interval).setMaxInterpolation($warped_interval) Comparing Method 1 and Method 2 & Calculation Outputs The below screenshot shows how Methods 1 & 2 arrive at the same output Note the difference in calculated values. The Methods reviewed in this post most closely capture the true time-weighted standard deviation per day across the 4 signals. Caveats and Final Thoughts While this method is still the most mathematically correct, there is a slight loss in data at the edges. When combining the data in the final step, the beginning of $signal_2 falls at the end of $signal_1, and so on. There are some methods that could possibly address this, but this loss in samples should be negligible to the overall standard deviation calculation. This method is also heavy on processing, especially depending on the input signals' data resolution and as the overall number of signals being considered increases. It is most ideal to use this method if real-time results are not of high importance, and better fitting if the calculation outputs are input in an Organizer that displays the previous day's/week's/etc results.
    2 points
  14. Hello, Exports cannot currently be modified from the UI. That functionality will be available when R62 releases in the August-September time range. They can be modified manually through the API using the POST /export/id endpoint if you're comfortable with that, but I'd recommend the delete & recreate method for now. Thanks.
    2 points
  15. The challenge with specifying a shared folder by Path is that to the owner of the content, the workbook will show up in their own folder, while to the shared user it will show up in their Shared Folder. This means you'd have to specify a different path based on who was running the code. The easiest method would be to specify the workbook by ID. If you really need to specify by name, I'd suggest doing a spy.search or spy.workbooks.search to get the ID, and then use that as the argument in your push: Another alternative would be to have the workbook live in the Corporate folder. The docstring for spy.push shows how you can push to workbooks in the Corporate folder: f'{spy.workbooks.CORPORATE} >> MySubfolder >> MyWorkbook'
    2 points
  16. In Seeq's May 25 Webinar, Product Run and Grade Transition Analytics, we received a lot of requests during the Q&A asking to share more details about how the grade transition capsules were created and how they were used to build the histograms presented. So here is your step-by-step guide to building the condition that will make everything else possible! Defining your transition condition Step 1: Identifying product campaigns The "condition with properties" tool under "identify" can transform a string signal of product or grade code into a single capsule per campaign. In older versions of Seeq this same outcome can be achieved in formula using $gradeCodeStringSignal.toCondition('grade') Note: if you don't have a signal representing grade code, you're not out of luck, you can make one! Follow the general methodology here to search for set point ranges associated with different grades or combinations of set points that indicate a particular grade is being produced. Step 2: Create the transition condition encoded with the starting and ending product as a capsule property $t = $c.shrink(2min).inverse() //2 min is an arbitrary small amound of time to enable the inverse function to //capture gaps in the campaigns $t.removelongerthan(5min) // 5min is an arbitrarily small amount of time to bound calc (that is >2min * 2) .transform($capsule -> $capsule.setProperty('Transition Type', $pt.resample(1min).toscalars($capsule).first() + '-->' + $pt.resample(1min).toscalars($capsule).last())) .shrink(2min) Step 3: Identify time period before the transition Step 4: Identify time period after the transition Step 5: Optional - for display purposes only - Show the potential transition timeline (helpful for chain view visualizations) Step 6: Create a condition for on-spec data Here we use value search against spec limit signals. If you don't have spec limit signals already in Seeq, you can use the splice method discussed in the post linked in step 1 to define them in Seeq. Step 7: Combine condition for on-spec data and condition for time period before the transition Step 8: Combine condition for on-spec data and condition for time period after the transition Step 9: Join the end of the condition for on-spec before the transition to the start of the condition for on-spec after the transition to identify the full transition duration Your condition for product transitions is now officially ready for use in downstream applications/calculations! Working with your transition condition Option 1: Calculate and display the transition duration Use Signal from Condition to calculate the duration. The duration timestamp will generate a visual where the length of the line is proportional to the length of the transition. Increasing the line width in the customize panel will display the transition duration KPI in a gantt-like view. Option 2: Build some histograms Summarize a count of capsules like the number of times each distinct transition type occurred within the date range. Summarize a signal during each unique transition type, like the max/min/average transition duration for a distinct transition type. Option 3: Generate some summary tables Switch to the table view and toggle "condition" to view each transition as a row in the table with key metrics like total duration and viscosity delta between the two grades. Add the "transition type" property from the "columns" selector. If you run into hiccups using this methodology on your data and would like assistance from a Seeq or partner analytics engineer please take advantage of our support portal and daily office hours.
    2 points
  17. Chromatography Transition Analysis in Seeq Many biopharmaceutical companies use Transition Analysis to monitor column integrity. Transition Analysis works by using a step change in the input conductivity signal and tracking the conductivity at the outlet of the column. The output conductivity transition can be analyzed via moment analysis to calculate the height of a theoretical plate (HETP) and the Asymmetry factor as described below. Step 1: Load Data and Find Transition Periods In order to perform this analysis in Seeq, we start by loading outlet conductivity and flow rate data for the column: Note: Depending on the density of the conductivity data, many companies find that some filtering of the data needs to be performed to get consistent results when performing the above differential equations. The agilefilter operator in Formula can be helpful to perform this filtering if needed: $cond.agileFilter(0.5min) Once the data is loaded, the next step is to find the transition periods. The transition periods can be found using changes in the signal such as a delta or derivative with Value Searches have also been applied. Combine the periods where the derivative is positive and the flow is low to identify the transitions. Alternative methods using the Profile Search tool can also be applied. Step 2: Calculate HETP As the derivatives are a function of volume instead of time, the next step is to calculate the volume using the following formula: Volume = $flow.integral($Trans) The dC/dV function used in the moment formulas can then be calculated: dCdV = runningDelta($Cond) / runningDelta($vol) Using that function, the moments (M0 through M2) can be calculated: M0 = ($dCdV*runningDelta($vol)).aggregate(sum(), $Trans, middleKey()) M1 = (($vol*$dCdV)*runningDelta($vol)).aggregate(sum(), $Trans, middleKey()) M2 = (($dCdV*($vol^2))*runningDelta($vol)).aggregate(sum(), $Trans, middleKey()) The moments are then used to calculate the variance: Variance = ($M2/$M0) - ($M1/$M0)^2 Finally, the HETP can be calculated: HETP = ((columnlength*$variance)/($M1/$M0)^2) In this case, the column length value needs to be inserted in the units desired for HETP (e.g. 52mm). The final result should look like the following screenshot: Alternatively, all of the calculations can be performed in a single Formula in Seeq as shown in the code below: $vol = $flow.integral($Trans) $dCdV = runningDelta($cond) / runningDelta($vol) $M0 = ($dCdV*runningDelta($vol)).aggregate(sum(), $Trans, middleKey()) $VdCdV = $vol*$dCdV $M1 = ($VdCdV*runningDelta($vol)).aggregate(sum(), $Trans, middleKey()) $V2dCdV = $dCdV*$vol^2 $M2 = ($V2dCdV*runningDelta($vol)).aggregate(sum(), $Trans, middleKey()) $variance = ($M2/$M0) - ($M1/$M0)^2 (52mm*$variance)/(($M1/$M0)^2) //where 52mm is the column length, L Step 3: Calculate Asymmetry Asymmetry is calculated by splitting the dC/dV peak by its max value into a right and left side and comparing the volume change over those sides. This section assumes you have done the calculations to get volume and dC/dV calculated already as performed for HETP in Step 2 above. The first step for Asymmetry is to determine a minimum threshold for dC/dV to begin and end the peaks. This is often done by calculating a percentage of the difference between the maximum and minimum part of the transition period (e.g. 10%): $min = $dCdV.aggregate(minValue(), $Trans, durationKey()) $max = $dCdV.aggregate(maxValue(), $Trans, durationKey()) $min + 0.1*($max - $min) The Deviation Search tool can then be used to identify the time when dC/dV is greater than the 10% value obtained above. Next, the maximum point of the dC/dV peaks can be determined by calculating the derivative of dC/dV in the Formula tool: $dCdV.derivative() The derivative can then be searched for positive values (greater than 0) with the Value Search tool to identify the increasing part of the dC/dV curve. Finally, a Composite Condition intersecting the positive dC/dV derivative and the transition values above 10% of the curve will result in the identification of the left side of the dC/dV curve: The right side of the dC/dV curve can then be determined using Composite Condition with the A minus B operator to subtract the positive dC/dV derivative from the transition above 10%: The change in volume can then be calculated by aggregating the delta in volume over each side of the peak using Formula: $Vol.aggregate(delta(), $LeftSide, middleKey()).aggregate(maxValue(), $Trans, middleKey()) Finally, the Asymmetry ratio can be calculated by dividing the volume change of the right side of the peak divided by the volume change of the left side of the peak. $VolRightSide/$VolLeftSide The final view should look similar to the following: Similar to HETP, all of the above formulas for Asymmetry may be calculated in a single formula with the code below: $vol = $flow.integral($Trans) $dCdV = (runningDelta($cond) / runningDelta($vol)).agileFilter(4sec) //calculate 10%ile of dCdV $min = $dCdV.aggregate(minValue(), $Trans, durationKey()) $max = $dCdV.aggregate(maxValue(), $Trans, durationKey()) $dCdV10prc = $min + 0.1*($max - $min) //Deviation search for when dCdV is above the 10%ile $deviation1 = $dCdV - $dCdV10prc $Above10 = valueSearch($deviation1, 1h, isGreaterThan(0), 0min, true, isLessThan(0), 0min, true) //Calculate filtered derivative of dCdV $dCdVderiv = $dCdV.derivative() //Value Search for Increasing dCdV (positive filtered derivative of dCdV) $dCdVup = $dCdVderiv.validValues().valueSearch(40h, isGreaterThan(0), 30s, isLessThanOrEqualTo(0), 0min) //Composite Conditions to find increasing left side above 10% and right side $LeftSide = $Above10.intersect($dCdVup) $RightSide = $Above10.minus($dCdVup) //Find change in volume over left side and right sides, then divide b/a $VolLeftSide = $Vol.aggregate(delta(), $LeftSide, middleKey()).aggregate(maxValue(), $Trans, middleKey()) $VolRightSide = $Vol.aggregate(delta(), $RightSide, middleKey()).aggregate(maxValue(), $Trans, middleKey()) $VolRightSide/$VolLeftSide Optional Alteration: Multiple Columns It should be noted that oftentimes the conductivity signals are associated to multiple columns in a chromatography system. The chromatography system may switch between two or three columns all reading on the same signal. In order to track changes in column integrity for each column individually, one must assign the transitions to each column prior to performing the Transition Analysis calculations. Multiple methods exist for assigning transitions to each column. Most customers generally have another signal(s) that identify which column is used. This may be valve positions or differential pressure across each column. These signals enable a Value Search (e.g. “Open” or “High Differential Pressure”) to then perform a Composite Condition to automatically assign the columns in use with their transitions. Alternatively, if no signals are present to identify the columns, the columns can be assigned manually through the Custom Condition tool or assigned via a counting system if the order of the columns is constant. An example of Asymmetry calculated for multiple columns is shown below: Content Verified DEC2023
    2 points
  18. Quick method to calculate the median of multiple related signals over a time period $allSignals = combinewith($p1fr.move(1ns), $p2fr.move(2ns), $p3fr.move(3ns), $p4fr.move(4ns)) $allSignals.aggregate(median(),hours(),durationkey()) The first part of the formula combines all the samples from your signals of interest into a new single signal which has all of the datapoints from the first four signals. One trick with the combinewith() function is that it can't combine samples that occur at the exact same timestamp. Since I am using data from a csv file for this example where the timestamps for all 4 signals are exactly the same I use the move() function to slightly offset each of the signals by a nanosecond so that they all combine smoothly. The final part of the formula is the aggregation where we can calculate the median value of our new combined signals which has all the sample form our prior 4 signals. In this example I am calculating an hourly median but this can be changed to daily() or a calculated condition that makes the most sense for your application.
    1 point
  19. Hi Pat, I think "$condition" in the formula is the parameter you are searching for: In Data Lab you should be able to access it via the "Formula Parmeters" column of the data frame: Regards, Thorsten
    1 point
  20. Fantastic, @Manoel Janio Borralho dos Santos! Great job using the SPy documentation examples as a reference for this use-case. Glad this worked out well for you.
    1 point
  21. Hi Jacob, You currently cannot export the table itself, but most table results you can recreate as a calculation and then export those items via OData. OData works with signals or conditions. With OData, you can use the auto-update option and then schedule the refresh rate in PowerBI and it will always grab the last X days/weeks/months/etc. of data.
    1 point
  22. Hey Ivan, It looks like this is a bug. I will log it and tag you so you receive updates on status. Thanks! Mark
    1 point
  23. I have a signal for the production LAST hour. I want to create "Hourly production" from this. Since the samples in the raw data (below image) is not always right on the whole hour I want to use the value from this signal on the half hours and shift that value 90 minutes back to the previous whole hour. So I want the value of the step signal below from 14:30 to be placed at exactly 13:00 in my new signal. And 15:30 -> 14:00 and so on. How would i go about that?
    1 point
  24. If I'm understanding you correctly, you want to find the last N batches that were ran that are of the same product as the most recent batch, correct? We can do that with some more creative condition logic and filtering 🙂 Assuming that our batch capsules have a property called 'Product' that tells us what product that batch ran, we can add some filtering to our above formula to only pass in capsules to toCapsulesByCount() that match the last product produced: // return X most recent batches in the past Y amount of time $numBatches = 3 // X $lookback = 3mo // Y $currentLookback = capsule(now()-$lookback, now()) // find the product of the most recent batch and assign that // as a capsule property to the lookback condition // this will let us filter the batches condition dynamically $lookbackCond = condition($lookback, $currentLookback) .setProperty( 'Product', $batchCondition.toSignal('Product', endKey()), endValue(true) ) // filter the batch condition to only find batches of the active product $filteredBatches = $batchCondition.removeLongerThan($lookback) .touches($lookbackCond, 'Product') // create a rolling condition with capsules that contain X adjacent capsules $rollingBatches = $filteredBatches.toCapsulesByCount($numBatches, $lookback) // find the last capsule in the rolling condition that's within the lookback period $batchWindow = condition( $lookback, $rollingBatches.toGroup($currentLookback, CAPSULEBOUNDARY.ENDSIN).last() ) // find all the batches within the capsule identified // ensure all the batches are within the lookback period $filteredBatches.inside($batchWindow) .touches(condition($lookback, $currentLookback)) The key here is to pass a capsule property name to touches() to allow filtering with this dynamic property value (the keep() function requires the input comparison scalar to be certain). The output will look something like this -- note I'm only getting the 3 more recent batches so you can actually read the property labels 🙂
    1 point
  25. One thing to add to this is SPy has been upgraded since this post was made to allow more options of templatizing content, including Organizers with images! Take a look at the documentation for Templates with images here, and the base documentation for Templates here. This serves a different approach to replacing images in Organizer that doesn't require dealing directly with the HTML.
    1 point
  26. Hi Koki, Please do book a slot in our Open Office hour, https://info.seeq.com/office-hours, so that you can share your Seeq screen and we can brainstorm from there. Thank you. Regards, Sharlinda
    1 point
  27. Hi KOKI, Generally for both #2 and #3, we often will bring the data in as capsules with capsule properties. It ultimately depends on how your system reports it, but typically the data is often reported as a time series of when the test is run (e.g. when the material property is recorded or quality data is run) and/or when the material is used (e.g. when you add a material to the batch, it will record what lot number you used and how much). Obviously it depends a bit on what and when your data is recorded, but Seeq has a number of tools to move the timeseries to align with when you want them as long as there is some logic that we can attach to (such as X batch ID had Y material number, which then had Z property value. There's some examples of this time alignment available at the article below - you specifically may want to look at #4:
    1 point
  28. Hi Marcelo, I am not certain I understand but I think you want to count the number of regenerations in between each "cleanup". If this is the case then you likely need to define a condition for when "cleanups" actually occur, and then you can create a condition for "time in between cleanups" using the .inverse() Formula function (there are several relatively easy you might find the "time in between cleanups"). Once you have the condition for "time in between cleanups", then a runningSum() Formula function would be one way to count the current number of regenerations. I'm happy to try to help iterate with your solution here on the forum, but because I'm not certain about the calculations you want to do, the most efficient way to get a solution for your questions would be to sign up for an upcoming Seeq Office Hours where you can share your screen and get help: https://info.seeq.com/office-hours John
    1 point
  29. Hello, yes this is possible and you can do it in different ways: You can use the Union Operator to unify all these condition into a single one then calculate the average for each capsule. The Formula tool is useful here and forumula should look like: $signal.aggregate(average(), $condition1.union($condition2).union($condition3).union($condition4).union($condition5), startKey()) You can also use formula and the splice operator to achieve similar result: $signal.aggregate(average(), $condition1, startKey()).splice($signal.aggregate(average(), $condition2, startKey()), $condition2).splice($signal.aggregate(average(), $condition3, startKey()), $condition3).... With one of the above methods, you will get the average of your signal for each capsule of the conditions 1 to 5. Finally, as you want to calculate the cumulative average, you can apply the runningsum() formula to the result of the previous step. Let me know if this is helpful.
    1 point
  30. Data Lab allows for multiple jobs to be scheduled in the same project. Each job should create a separate executor.0.html file in the _Job Results folder after the job has run. Are you sure that your first job was scheduled correctly? The attached screenshots don't show the full schedule confirmation & detail so it is hard for me to tell. If you are still having issues, feel free to create a support ticket at https://support.seeq.com/ so we can troubleshoot further!
    1 point
  31. Hi Tayyab - One alternative would be to disable inheritance on a folder in the Corporate Drive and then removing the "Everyone" group. To do so, create a folder in the Corporate Drive, click the "three dots" on the right, and select "Manage Permissions". You will see that "Everyone" has access. Click on "Advanced" and "Disable permission inheritance". You can now remove the "Everyone" group from your folder access and add your collaborators and have a folder all your collaborators share (and not "Everyone").
    1 point
  32. Sorry, try this: # Pull the workbook workbooks = spy.workbooks.pull(workbook_id) workbook = workbooks[0] # Add the template worksheet if template_worksheet.name in workbook.worksheets: workbook.worksheets[template_worksheet.name] = template_worksheet else: workbook.worksheets.append(template_worksheet) # Push back up spy.workbooks.push(workbook)
    1 point
  33. Wow.. I was overthinking this way too much 😅 Thanks John I appreciate the help!
    1 point
  34. Simple math operations like subtracting signals can be easily done with the Formula tool: In this case to find the deviation from SP you could do the following:
    1 point
  35. OK Pat here is the fixed-up ipynb file. Do a search for "MarkD says" and you'll see how I've left comments to explain what changes I made. topic_document_deprecated (1).ipynb
    1 point
  36. When attempting to install Seeq version R60.1.1, I received the error depicted in the screenshot when executing the "from seeq import spy" command. I successfully attempted to uninstall the currently installed version of Seeq to match the server version but couldn't import spy. When switching back to Seeq version R58.1.1, I was able to run this line without error. Is this a bug related to R60? Thanks, Bao
    1 point
  37. Looking back, you're correct that R57+ does not have the ability to configure a scheduled update every X months. Could you submit a support ticket and include the link to this post: https://seeq.atlassian.net/servicedesk/customer/portal/3? That way we can log the change and see if we can get you a workaround in the meantime.
    1 point
  38. Are you just looking for an hourly average of your signal, or are you looking to do averages over more complex conditions? If you're looking for hourly averages, you can use the following as an input to the calculation parameter: spy.pull( items, start='01/01/2023', calculation='$signal.aggregate(average(), days(), startKey())', grid=None ) Another way to accomplish this is to include both a signal and a condition in your call to spy.pull and specify shape='capsules'. This will result in a dataframe where rows are capsules, and columns are aggreaged signals. This can be good when you want to use more complex conditions to define your aggregations:
    1 point
  39. Hello Bao, When upgrading to r60+ you will need to use spy.upgrade() instead of the pip install seeq commands. Please let me know if this works for you. spy.upgrade() Regards, Rupesh
    1 point
  40. Hi Anton, Thank you for the question. Unfortunately, there is not currently a supported method for explicitly pulling a table view from Seeq WB into SDL. Depending on what you have in the table, you may be able to get it pretty close with a typical spy.pull using the shape and or capsule_properties arguments. That said, there are certainly instances where the spy.pull alone will not get the table you are looking for. You could leverage pandas to manipulate the table to the format you are looking for as well. I hope this helps. Thank you, Synjen
    1 point
  41. Hi Emre, you could try to sum up the running deltas for each months: $signal.setMaxInterpolation(20d).runningdelta().runningSum(months('CET')) Regards, Thorsten
    1 point
  42. Hi Johannes, there is no direct way to do this in workbench, but you can use the API to do this. I would suggest testing this first, before applying this to your condition. More information can be found in this post: Regards, Thorsten
    1 point
  43. An oil and gas engineer would like to reproduce below Batch monitoring table for vessels in Seeq. Green color indicates that the valve for the respective batch step is open, while red color indicates the valve status is close. 1. Use Condition with Properties to create conditions with regen steps as property name. 2. Define the respective valves position at each regen step. $Running = ($Prod_Valve ~= "Open").intersect($Step.keep('Batch Step',isEqualTo('RUNNING'))) $Isopropyl = ($Prod_Valve ~= "Open").intersect($Step.keep('Batch Step',isEqualTo('ISOPROPYL FLUSH'))) $Startup = ($Prod_Valve ~= "Open").intersect($Step.keep('Batch Step',isEqualTo('STARTUP'))) $Down = ($Prod_Valve ~= "Close").intersect($Step.keep('Batch Step',isEqualTo('DOWN'))) combineWith ($Running,$Isopropyl,$Startup,$Down) 3. Create a scorecard metric with colour threshold for each valve: Count =1; Green – good with single valve opening Count >1 or <1; Red – alarm as potential multiple valves opening or issue with signal status 4. In condition Table view, select the Capsule Property at the Headers. Optionally, the count value can be hidden by editing "" in the Number Format in the Item Properties as shown below. With this method, if all the valves indicates green means the valve Open/Close correctly. Thus user just need to focus on the red indicator.
    1 point
  44. Hi Margarida, you can use the following settings in the request body: { "unitOfMeasure": "string", "value": "{\"type\":\"formula\",\"advancedParametersCollapsed\":true,\"helpShown\":false}" } If you want to switch back to the UI tool you should save your previous value somewhere, so you could reuse this in the POST request. Regards, Thorsten
    1 point
  45. Hi, The radar/spider/polar plot is not a native Seeq visualization. That said, you could use one of the many python libraries (matplotlib, Seaborn, plotly)out there that supports this type of plot to create the visual in Seeq Data Lab(SDL). The visual you create in SDL can be pushed to an Organizer Topic as an image. This workflow can also be automated to push an updated plot as needed. Simple example: #save plot as png once created plot.savefig('chart.png') #Use spy workbooks to find your topic/sheet topic = spy.workbooks.Topic({'Name': "Charts Testing Push"}) page = topic.document('Visualizations Using Data Lab') #add image to topic page.document.add_image(filename='chart.png', placement='end') #push topic iwth new image spy.workbooks.push(topic) Here is a post on replacing images using a more advanced method with html: To automate the workflow you can schedule the notebook that creates and pushes the plot to organizer using spy.jobs.
    1 point
  46. Seeq is able to do many time periods, but what if you want to compare a time period from now, with similar periods in the past? An example of this could be comparing the 24 hour period before now, with the previous 24 hour period, and the one before etc. Add in that you want this period to be rolling and it becomes more complex. Example: The time now is 3PM, you want to look at the period of yesterday 3PM till today 3PM, and compare with similar 24 hour periods over the last 30 days. Then you want to do it again every 5 minutes with up to date time periods, so 3:05PM with the pervious 24 hours and similar periods for the last 30 days. The way to do this involves several steps, but basically the solution is to look at the time now from Midnight past, then move a daily periodic condition by this time, which is ever changing. Once you have executed this formula you can use the resulting capsules for aggregations on the rolling time periods extending into the past. You can paste the following formula directly into Seeq Formula tool, no need for any signals to be added. $days = days("Europe/London") $daynow = condition(1d, capsule(now()-24h, now())) //isolate the current daily capsule $timesince = timeSince($days, 1min) //time since midnight in minutes $timesincediscrete = $timesince.aggregate(endValue(true), $daynow, endKey(), 40h) //make discrete //Now going to use the discrete data point for time since midnight to make that time go into the past to cover // every daily capsule that is going to be moved in time, I am using 30 days, but change this to cover your own use case $movetimesincetopast = $timesincediscrete.move(-30 days) //now to join the current timesince midnight to the one moved into the past //we also have to resample the result to ensure data points are available for every capsule. The maxinterpolation should be greater //than your timescale, in my case greater than the 30 days used above $combined = $timesincediscrete.combineWith($movetimesincetopast).setMaxInterpolation(40 days).setunits('mins') $combinedresampled = $combined.resample(1min) //Now to transform our daily capsules to move the start of the capsules by the current time from midnight, note we can //only move the start as if we also move the end it will result in an error as the resulting capsules would no longer //be contained by the original capsules $transform = $days .transform($capsule -> { $newStartKey = $capsule.startKey() + $combinedresampled.valueAt($capsule.startKey()) capsule($newStartKey, $capsule.endKey()) }) //Finally we can use the start of the transformed capsules to create new 24 hour capsules, these will continually update on a refresh, or update frequency assigned in workbench //Use the 'step to current time' button in the display pane to see the start/end times of the capsules update in the capsule pane $finalcapsules = $transform.afterstart(24 hours) Return $finalcapsules Notes: If you extend this over a large time period/very high frequency refresh, you will cause significant calculations to be performed, slowing your Seeq system The resulting capsules will always be uncertain, as the time period is ever changing. Whilst I have used 'Days' & '24h', you can alter the formula for any time periods, but bear in mind that very short periods will cause significant calculations if over a relatively long time period into the past.
    1 point
  47. We often get asked how to use the various API endpoints via the python SDK so I thought it would be helpful to write a guide on how to use the API/SDK in Seeq Data Lab. As some background, Seeq is built on a REST API that enables all the interactions in the software. Whenever you are trending data, using a Tool, creating an Organizer Topic, or any of the other various things you can do in the Seeq software, the software is making API calls to perform the tasks you are asking for. From Seeq Data Lab, you can use the python SDK to interact with the API endpoints in the same way as users do in the interface, but through a coding environment. Whenever users want to use the python SDK to interact with API endpoints, I recommend opening the API Reference via the hamburger menu in the upper right hand corner of Seeq: This will open a page that will show you all the different sections of the API with various operations beneath them. For some orientation, there are blue GET operations, green POST operations, and red DELETE operations. Although these may be obvious, the GET operations are used to retrieve information from Seeq, but are not making any changes - for instance, you may want to know what the dependencies of a Formula are so you might GET the item's dependencies with GET/items/{id}/dependencies. The POST operations are used to create or change something in Seeq - as an example, you may create a new workbook with the POST/workbooks endpoint. And finally, the DELETE operations are used to archive something in Seeq - for instance, deleting a user would use the DELETE/users/{id} endpoint. Each operation endpoint has model example values for the inputs or outputs in yellow boxes, along with any required or optional parameters that need to be filled in and then a "Try it out!" button to execute the operation. For example, if I wanted to get the item information for the item with the ID "95644F20-BD68-4DFC-9C15-E4E1D262369C" (if you don't know where to get the ID, you can either use spy.search in python or use Item Properties: https://seeq.atlassian.net/wiki/spaces/KB/pages/141623511/Item+Properties) , I could do the following: Using the API Reference provides a nice easy way to see what the inputs are and what format they have to be in. As an example, if I wanted to post a new property to an item, you can see that there is a very specific syntax format required as specified in the Model on the right hand side below: I typically recommend testing your syntax and operation in the API Reference to ensure that it has the effect that you are hoping to achieve with your script before moving into python to program that function. How do I code the API Reference operations into Python? Once you know what API endpoint you want to use and the format for the inputs, you can move into python to code that using the python SDK. The python SDK comes with the seeq package that is loaded by default in Seeq Data Lab or can be installed for your Seeq version from pypi if not using Seeq Data Lab (see https://pypi.org/project/seeq/). Therefore, to import the sdk, you can simply do the following command: from seeq import sdk Once you've done that, you will see that if you start typing sdk. and hit "tab" after the period, it will show you all the possible commands underneath the SDK. Generally the first thing you are looking for is the ones that end in "Api" and there should be one for each section observed in the API Reference that we will need to login to using "spy.client". If I want to use the Items API, then I would first want to login using the following command: items_api = sdk.ItemsApi(spy.client) Using the same trick as mentioned above with "tab" after "items_api." will provide a list of the possible functions that can be performed on the ItemsApi: While the python functions don't have the exact same names as the operations in the API Reference, it should hopefully be clear which python function corresponds to the API endpoint. For example, if I want to get the item information, I would use "get_item_and_all_properties". Similar to the "tab" trick mentioned above, you can use "shift+tab" with any function to get the Documentation for that function: Opening the documentation fully with the "^" icon shows that this function has two possible parameters, id and callback where the callback is optional, but the id is required, similar to what we saw in the API Reference above. Therefore, in order to execute this command in python, I can simply add the ID parameter (as a string as denoted by "str" in the documentation) by using the following command: items_api.get_item_and_all_properties(id='95644F20-BD68-4DFC-9C15-E4E1D262369C') In this case, because I executed a "GET" function, I return all the information about the item that I requested: This same approach can be used for any of the API endpoints that you desire to work with. How do I use the information output from the API endpoint? Oftentimes, GET endpoints are used to retrieve a piece of information to use it in another function later on. From the previous example, maybe you want to retrieve the value for the "name" of the item. In this case, all you have to do is save the output as a variable, change it to a dictionary, and then request the item you desire. For example, first save the output as a variable, in this case, we'll call that "item": item = items_api.get_item_and_all_properties(id='95644F20-BD68-4DFC-9C15-E4E1D262369C') Then convert the output "item" into a dictionary and request whatever key you would like: item.to_dict()['name']
    1 point
  48. Here is an example of Scatter plot push. @Asset.Display() def Display(self,metadata,analysis): #define name of worksheet worksheet = analysis.worksheet(self.definition['Name']+' Scatter') #define name of workstep workstep = worksheet.workstep('Trends') # make sure workstep is in Scatter Plot view workstep.view = 'Scatter Plot' # define the display items - need to have at least the 2 signals you want in scatterplot workstep.display_items = [{ 'Item': self.Temperature(metadata) },{ 'Item': self.Relative_Humidity(metadata) }] # define the 2 scatterplot signals workstep.scatter_plot_series = { 'X': {'ID': self.Temperature(metadata)}, 'Y': {'ID': self.Relative_Humidity(metadata)} } return workstep #Thanks @Kjell Raemdonck for the code snippet
    1 point
  49. As a Seeq user, you may have created a condition for a particular event of interest and would like to create a signal that is the running count of these events over a given time period. This analysis is common in equipment fatigue use cases when equipment degrades slowly based on a number of cycles (thermal, pressure, tension, etc) that it has undergone during it's life or since a last component replacement. This use case can be done very efficiently in Seeq Formula. The assumptions for the below solution are: You have a condition ($condition) of interest that you would like to understand the running count for There is a defined timeframe of interest, where counting will start and end. Note the end date can be sometime in the future. For the below example, this condition is referenced as $manualCondition, but could very well be another condition that wasn't created via the Manual Condition tool. Just note that for each capsule in this condition, the count will restart at 0. Solution - Utilize the runningCount() formula function: 1) runningCount() currently only accepts signals as inputs, so convert your $condition to a signal by using .toSignal(), which produces a single sample for each capsule: $condition.toSignal(SAMPLE_PLACEMENT) SAMPLE_PLACEMENT should be specified as startKey(), middleKey(), or endKey(). If you want your count to increase at the start of each event, then use startKey(). If wanting the count to increase in the middle or end of each event, then use middleKey() or endKey() 2) Use the runningCount() function on the signal created above. $signal.runningCount($conditionToStartAndEndCounting) Both steps are shown below in a unified Formula: /* This portion yields a SINGLE point for each capsule. startKey() generates the point at the START of each capsule, where middleKey() and endKey() could also be used to generate a point at the MIDDLE or END of each capsule. Where these points are placed matter, as that is the point in time the count will increase. */ $samplePerCapsule = $condition.toSignal(startKey()) /* This portion yields the running count of each sample (capsule). The 15d in toStep() can be adjusted. Ideally this number will be the duration of the longest expected time between two events that are being counted. */ $samplePerCapsule.runningCount($manualCondition).toStep(15d) .toStep(15d) ensures the output signal is step interpolated, interpolating points at most 15 days apart. If step interpolation is not required, then this can be removed or replaced with something like .toLinear(15d). Below shows the associated output. Content Verified DEC2023
    1 point
  50. The "!" operator for "not" is also now supported in Seeq Formula. The same condition that Morgan created above can be returned by: !$s2 More information on this syntax can be found in the Formula Documentation on the inverse operator.
    1 point
This leaderboard is set to Los Angeles/GMT-07:00
×
×
  • Create New...