Identify subtle trends or step changes in a signal

[Allison Buenemann]

Background: 

When looking to identify trends or step changes in a signal, we typically recommend an approach of smoothing the signal, taking the first derivative, then identifying when that derivative is positive or negative. This method works well most of the time, but employing this technique in combination with others can be more effective at capturing trends/step changes when the value change in the signal is more subtle. 

Solution:

When looking for step changes, we can use a technique of calculating a range of the signal on a rolling periodic basis and search for when the range exceeds some limit. We can then combine this condition with when the derivative is positive (increasing step changes) or negative (decreasing step changes) to capture our final condition. 

1. Create a rolling window over which you will look at the range (max-min value) of the signal. In my example I used a 4h window every 30 minutes, because my tank draining events were typically never longer than 4h. Select the smallest time period that you can that is still longer than your longest draining event.

2. Use Signal from Condition to calculate the range (max-min) of your signal over each of the rolling windows. Make sure to place the time stamp of the statistic at the end of each rolling capsule. 

3. Identify time periods when that range calculation is above some threshold. In this example we used a threshold of 2 based looking at the trend output of our step 2. 

If we zoom in on a smaller range of time, we see that our capsules for when the range value is high actually extend beyond the completion of our decreasing signal. 

4. We can intersect this condition that we have identified for high range in the signal with a condition for when the derivative of the signal is negative to capture our desired events. First calculate the first derivative of the signal. We apply a smoothing agileFilter in this step as well to remove signal noise.

5. Identify when that derivative value is less than zero using the value search tool.

6. Now take the intersection of the condition for negative derivative of the level and the condition for high range.

The final view of the original signal and the events identified:

Use chain view to validate your calculations: