Clean Signal by Using Percentile Calculation in Training Range

[Shamus Cunningham]

This is a solution for a question that came in the support channel that I though would be of general interest. The question was how to designate a fixed training range for a signal and then calculate upper and lower limits of the signal using the 3rd and 97th percentile and apply those limits to the entire history of the signal 

This requires a two step process. The first is to create scalar signals for the upper and lower limits. Next we use those upper and lower limits to clean the signal using the remove() formula 

Step 1) Calculating the Scalar values for the 97th and 3rd Percentiles 

In the example below the training range start and end dates are hard coded into the formulas for simplicity

 

$trainingRangeStart = '2022-10-01T00:00:00Z'
$trainingRangeEnd = '2022-10-31T00:00:00Z'

$trainingCondition = condition(capsule($trainingRangeStart,$trainingRangeEnd))

$calcPercentile = $signal.aggregate(percentile(97), $trainingCondition, startKey())

$calcPercentile.toScalars(capsule($trainingRangeStart,$trainingRangeEnd)).average()

Similar formula for the lower limit 

$trainingRangeStart = '2022-10-01T00:00:00Z'
$trainingRangeEnd = '2022-10-31T00:00:00Z'

$trainingCondition = condition(capsule($trainingRangeStart,$trainingRangeEnd))

$calcPercentile = $signal.aggregate(percentile(3), $trainingCondition, startKey())

$calcPercentile.toScalars(capsule($trainingRangeStart,$trainingRangeEnd)).average()

 

Step 2) Clean the signal using the new scalar values for upper and lower limits 

$signal
.remove(isGreaterThan($upper))
.remove(islessthan($lower))