Close

Introducing the De-Burr filter

A project log for analog.io - A full stack IoT platform

A full stack project dedicated to easily collecting, analyzing and sharing IoT sensor data.

luke-benoLuke Beno 09/14/2015 at 07:261 Comment

One common issue that has been observed with many IoT streams is that they often have what we're calling burrs or anomalies where the sensor measurement is erroneous and produces a dis-continuous and incorrect measurement.

Here is an example:

Burrs are especially inconvenient because of the way that analog.io currently manages the vertical scale of the data plot: They tend to hide the important information that you actually want to look at!

For this reason, we've added a de-burr filter to analog.io! It is disabled by default but the rest of this blog post will describe how to enable it.

The de-burr filter is accessible from the "Settings" menu within the stream. When you click the settings button, a dialog will appear as shown below.

To enable the de-burr filter, simply check the box and enter a coefficient. Wait! What is a coefficient?!?!

The the de-burr filter operation is pretty simple, it scans through the stream data and compares the most current value with the previous sample. The coefficient defines the acceptable range that one sample could vary from the previous. It is specified as a ratio of the current value. An example would be very helpful:

Let's say that we have a current value of 44 degrees and the next sample is a burr with a value of 202 degrees. If we set a coefficient of 0.5, this means that in order for the next sample to pass through the filter, it would need to be greater than 44-44*0.5 = 22 and less than 44+44*0.5=66. Since 202 is not less than 66, it will be removed from the stream. Good bye burr!

Let's try that same Temperature data with the de-burr filter on and the coefficient set to 0.5:

Note: After saving your settings, you will need to do a browser refresh for the changes to take effect

That looks much better, now we can see all of the wonderful details that were previously hidden by the burrs!

Discussions

Jose Ignacio Romero wrote 03/19/2018 at 19:35 point

Another way to handle this is to use an "interquartile mean", For that technique you take a set of samples, remove the top and bottom quartile and average the rest. For example, if you take 8 samples you throw away the highest 2 and lowest 2, then average the remaining 4, and you'll have an average that is resistant to outliers.

  Are you sure? yes | no