If you have been using Home Assistant to track any kind of consumption like Water, Gas, Electricity ETC, You have probably set up a few devices in the energy dashboard only to come back in a few days and be told you somehow used 900 kWh last night on a heater. Wait WHAT??? after looking at the devices history you notice a huge spike in the readings that then goes back to normal.

Well that’s frustrating! After doing some research on the Home Assistant Community Forum I found that the most likely cause of this issue is when a device is not reachable for some reason but its intagration or services does not return an proper unavailable state. In the energy dashboard if sees your device go from 3kWh -> unavailable -> to 3kWh it does nothing and ignores the unavailable. If it sees you go from 3kWh to any value other than a nubmer or unavailable, It will evaluate the value to a 0. Meaning the energy dashboard will see 3kWh -> 0kWh -> to 3kWh. The energy dashboard assumes you just used 3kWh, which is correct based on the values it received but is actually incorrect. The best solution to correct this issue it to make sure that either a nubmer value OR unavailable is reported to the energy dashboard. Problem is that its someone elses itagration, firmware or hardware that is causing.

Since we can’t change those items as they are out of our control, the next best thing you can do is to use the availability feature on a template sensor. What this does is allows you to ensure your sensor either reports a numeric value or unavailable. You achieve this by putting this template sensor in front of your source sensor and then put this template sensor in the energy dashboard. Please note the availability template features is not available in the GUI, you need to create the heper in your configuration.yaml file. This is a great solution if you are having the issue that your sensor is going off line or is not always reachable because a cloud service or wifi is down.

The availability template only factors this one issue, some sensors just give bad readings. For me I noticed that a few z-wave devices would just randomly add a random amount of kWh to the sensor and then go back. This was not like the issue above where the sensor when into an unknown state then returned. For these devices I wanted a way to filter the input to ensure it cannot change by more than a certain value. The first place I looked was the built in filters for Home Assistant. Sadly none of the filters deal with consumption only devices (Devices that should only go up in value, never down). The Outlier filter was a really close fit but there were two issues with it. When correcting an out of band reading, it allowed the consumption to go down because it pulls the median value of the last x readings when replacing a bad read. I tried to correct this by only looking at the last reading (set the window size to 1) this worked better and didn’t allow any negative consumption, it would only allow a reading equal to the last or above. The problem was, this introduced a different issue because the filter always pulls is values for replacement from the original sensor, so if there is a single bad reading it will detect it and replace it with the previous reading which is great, works as expected. BUT if there was a second bad reading it will replace that with the last value of the original sensor which is a bad reading. We are drinking from a poisoned well at that point. This is why the filter did not work.

So my only other option was to program an entire filter myself using custom components in Home Assistant. I did git it partially done but then thought, this is just the same as creating a sensor template with availability on it. Let’s save some time and just create one of those!!

I wanted to make sure I could continue to use the GUI to edit my helpers so I opted out of using the configuration.yaml. This means that I could not use the availability section as the availability section its not exposed to the GUI in Home Assistant (would be a nice feature to get YAML editor in GUI for template devices). So to get around this I decided to code the availability into the template section as it really is the same thing.. Return unavailable if one of the devices is not around! I added a variable max_jump to state what the largest jump this devices can make in consumption before it assumes its a bad reading and replaces it with the last good reading (and here it the part that makes it work) from the filtered side of the sensor. This way you are only replacing bad readings with filtered readings. You will need to run a device for a few days so that you can get a good gauge on how much your device consumes between readings. Please also remember there is a possibility that the wifi or this devices cloud could or z-wave controller could go down. Consumption monitoring devices will still continue to record their consumption, it just can’t report it to HA. Once that services comes back online the consumption devices will report and that reading could be a big jump if it was unavailable for few hours. So I will need to set the max_jump to something with breathing room. IE if on average in a normal week this device only consumes 0.01kWh between readings then I would make it 0.5 as this gives it some space to allow for it to be offline for some time or it happens to consume 10x more than expected because it goes through a defrost cycle every so often.. If you set this value too low then there is a possibility this filter would stop functioning as the real reading will be right but your filter will always be above the last state and will get filtered out. So be careful, but if this does happens just set max_jump to a really large nubmer to allow the next read in then set it back to something better!

Here is the code:
{% set max_jump = 0.1 %}
{% set raw = states(‘sensor.123_test_sensor’) %}
{% set last_state = this.state %}
{% set raw_invalid = raw in [‘unknown’, ‘unavailable’, ‘none’, ”] %}
{% set last_invalid = last_state in [‘unknown’, ‘unavailable’, ‘none’, ”] %}

{% if raw_invalid and last_invalid %}
unavailable
{% elif raw_invalid %}
{{ last_state }}
{% elif last_invalid %}
{{ raw | float }}
{% else %}
{% set current = raw | float %}
{% set last = last_state | float %}
{% if (current – last) | abs <= max_jump %}
{{ current }}
{% else %}
{{ last }}
{% endif %}
{% endif %}

If you have any improvements of can find an issue with it please let me know! I want to make this as hardened as possible for edge cases like meter resets, HA reboots.. If I am missing something I would be glad to fix the code!