Aggregated Time Weighted Standard Deviation Across Multiple Signals

[Emilio Conde]

There are times when you may need to calculate a standard deviation across a time-range using the data within a number of signals. Consider the below example.

 

When a calculation like this is meaningful/important, the straightforward options in Seeq may not be mathematically representative to calculate a comprehensive standard deviation. These straightforward options include:

• Take a daily standard deviation for each signal, then average these standard deviations
• Take a daily standard deviation for each signal, then take the standard deviation of the standard deviations
• Create a real-time standard deviation signal (using stddev($signal1, $signal2, ... , $signalN)), then take the daily average or standard deviation of this signal

While straightforward options may be OK for many statistics (max of maxes, average of averages, sum of totalizes, etc), a time-weighted standard deviation across multiple signals presents an interesting challenge. 

This post will detail methods to achieve this type of calculation by time-warping the data from each signal then combining each individually warped signal into a single signal. Similar methods are also discussed in the following two seeq.org posts:

 

 

Two different methods to arrive at the same outcome will be explored. Both of these methods share the same Step 1 & 2.

 

Step 1: Gather Signals of Interest

This example will consider 4 signals. The same methods can be used for more signals, but note that implementing this solution programmatically via Data Lab may be more efficient when considering a high number of signals (>20-30).

 

Step 2: Create Important Scalar Constants and Condition

1. Number of Signals: The number of signals to be considered. 4 in this case.
2. Un-Warped Interval: The interval you are interested in calculating a standard deviation (I am interested in a Daily standard deviation, so I entered 1d)
3. Warped Interval: A ratio calculation of Un-Warped Interval / Number of Signals. This metric is detailing what the new time-range will be for the time-warped signals. I.e. given I have 4 signals considering a days worth of data of, each signal's day worth of data will be warped into 6 hour intervals
4. Un-Warped Periods: This creates a condition with capsules spanning the original periods of interest. periods($unwarped_interval)

 

Method 1: Create ONE Time-Shift Signal, and move output Warped Signals

The Time Shift Signal will be used as a counter to condense the data in the period of interest (1 day for this example) down to the warped interval (6 hours for this example).

0-timeSince($unwarped_period, 1s)*(1-1/$num_of_signals)

 

The next step is to use this Time Shift Signal to move the data within each signal. Note there is an integer in this Formula that steps with each signal applied to. Details can be viewed in the screenshots.

$area_a.move($time_shift_signal, $unwarped_interval).setMaxInterpolation($warped_interval).move(0*$warped_interval)

 

The last step is to combine each of these warped signals together. We now have a Combined Output that can be used as an input into a Daily Standard Deviation that will represent the time-weighted standard deviation across all 4 signals within that day.

 

Method 2: Create a Time-Shift Signal per each Signal - No Need to move output Warped Signals

This method takes advantage of 4 time-shift signals, one per signal. Note there is also an integer in this Formula that steps with each signal applied to. Details can be viewed in the screenshot. These signals take care of the data placement, where-as the data placement was taken care of using .move(N*$warped_interval) above.

0*$warped_interval-timeSince($unwarped_period, 1s)*(1-1/$num_of_signals)

 

We can then follow Method 1 to use the time shift signals to arrange our signals. We just need to be careful to use each time shift signal, as opposed to the single time shift signal that was created in Method 1. As mentioned above, there is no longer a .move(N*$warped_interval) needed at the end of this formula. 

The last step is to combine each of these warped signals together, similar to Method 1. 

$area_a.move($time_shift_1, $unwarped_interval).setMaxInterpolation($warped_interval)

 

 

Comparing Method 1 and Method 2 & Calculation Outputs

The below screenshot shows how Methods 1 & 2 arrive at the same output

 

Note the difference in calculated values. The Methods reviewed in this post most closely capture the true time-weighted standard deviation per day across the 4 signals.

 

Caveats and Final Thoughts

While this method is still the most mathematically correct, there is a slight loss in data at the edges. When combining the data in the final step, the beginning of $signal_2 falls at the end of $signal_1, and so on. There are some methods that could possibly address this, but this loss in samples should be negligible to the overall standard deviation calculation.

This method is also heavy on processing, especially depending on the input signals' data resolution and as the overall number of signals being considered increases. It is most ideal to use this method if real-time results are not of high importance, and better fitting if the calculation outputs are input in an Organizer that displays the previous day's/week's/etc results.