Jump to content

Clean Signal by Using Percentile Calculation in Training Range


Recommended Posts

  • Super Seeqer

This is a solution for a question that came in the support channel that I though would be of general interest. The question was how to designate a fixed training range for a signal and then calculate upper and lower limits of the signal using the 3rd and 97th percentile and apply those limits to the entire history of the signal 

This requires a two step process. The first is to create scalar signals for the upper and lower limits. Next we use those upper and lower limits to clean the signal using the remove() formula 

Step 1) Calculating the Scalar values for the 97th and 3rd Percentiles 

In the example below the training range start and end dates are hard coded into the formulas for simplicity

 

$trainingRangeStart = '2022-10-01T00:00:00Z'
$trainingRangeEnd = '2022-10-31T00:00:00Z'

$trainingCondition = condition(capsule($trainingRangeStart,$trainingRangeEnd))

$calcPercentile = $signal.aggregate(percentile(97), $trainingCondition, startKey())

$calcPercentile.toScalars(capsule($trainingRangeStart,$trainingRangeEnd)).average()

Similar formula for the lower limit 

$trainingRangeStart = '2022-10-01T00:00:00Z'
$trainingRangeEnd = '2022-10-31T00:00:00Z'

$trainingCondition = condition(capsule($trainingRangeStart,$trainingRangeEnd))

$calcPercentile = $signal.aggregate(percentile(3), $trainingCondition, startKey())

$calcPercentile.toScalars(capsule($trainingRangeStart,$trainingRangeEnd)).average()

image.png

 

Step 2) Clean the signal using the new scalar values for upper and lower limits 

$signal
.remove(isGreaterThan($upper))
.remove(islessthan($lower))

image.png

  • Like 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...