Netatmo Webhooks are very unstable
I have several Netatmo cameras, both indoor and outside. They are all connected to Home Assistant, and I have established properly webhook connections using my Dev account.
The result is that sometimes I get webhook events, sometimes not. Sometimes I get all events, and at other times, noting at all. I haven't been able to find any pattern in this. It seems completely random. Everything is updated to the most resent versions. In the past week, I only get events everytime one of the cameras disconnect and reconnect, which can happen now and then. But it's been a week or so since I got any human detection events.
I need som advise on how to debug this.
Comments
29 comments
Hi Leslie,
i have the same issue here, 3 indoor cams 6 door sensors and 2 thermostats, and no netatmo_events on my Home Assistant, stooped as other user any months ago. Now sometimes i receive only one event of reconizeing faces but no other.
It' s a disaster for my autmations, i checked webhooks and configuration but the issue seems on netatmo side.
I m very sad for this becouse the products are not cheap and oriented to be integrated on other platform (home assistant), before working properly but since one day to other stopped..
Can you help me?
Hello,
Concerning the Doorbell webhooks, it seems that the incoming_call webhooks for 3rd parties "disappeared" for some reason (NDB-rtc was already supposed to be displayed, but they should arrive after incoming_call). The developers are warned and they opened an internal ticket for investigation. They'll warn me once they have more information
@oyvindo as this issue is different I think there is no link with the one you have. I checked our internal documentation and asked the teams, for them we don't have a webhook_id value. The only value to identify the endpoint is the redirect_uri. Don't you have any other way to put in place this monitoring ?
Have a good day,
Leslie - Community Manager
@leslie, thanks for your reply and sorry for not getting back sooner.
Meanwhile I've continued my research and debugging.
But to answer your questions first; Yes, I mean the dev/netatmo/apps console.
In my H+S app all events are recorded correctly in the timeline.
I have re-done the authorize process, but to no avail.
Now, here comes the funny part: I have two home Assistance instances; One is HA-OS based using NabuCasa and the other is HA-Container Core based and exposed to the internet through a Reverse Proxy.
These two instances are on two different networks in two different countries.
Both instances has the Netatmo integration and they both have indoor as well as outdoor cameras.
Webhooks and movement detection was working well on both up until sometime early this summer when I noticed that events happened less and less frequent, only to finally come to a complete halt on both systems - with one exception: Both systems still generate webhook events whenever a camera drops off-line, which do happen from time to time (once or twice each month, and it's always been like that).
But none of the cameras are able to provide event triggers for anything else; movements, human detection, alarms etc. I can run the Netatmo register/un-register service from both HA instances successfully. No errors reported.
Another side nuance; both outdoor cameras has lost the floodlight entity which for a while now is reported as unavailable. I have had no problems or instabilities with the video stream from any of the cameras (except when it drops off-line occasionally). If/when a camera does drop off-line, it normally is automatically restored again in less than an hour. Only rarely do I need to power cycle the camera to get it back on line.
I've never had a ban-situation in the dev/netatmo/app console for as long as I can recall.
The application credentials setting in HA on both instances are verified to be correct (remember - it does generate correct webhook events every time a camera drops off-line)
Rgds
Oyvind
Hello,
@francisco, I forwarded to the teams your message from your other post. We indeed had other feedbacks about notification delays, including reportings from Home + Security users (they mainly focus their investigation on the 16 September where they have detailed data)
@Oyvind, "In my H+S app all events are recorded correctly in the timeline." <= concerning the push notifications of the events, did you also correctly receive all of them ? (coming from Home + Security app) If no, it's a webhook issue on our side. If yes, something is blocking the notifications sent to your endpoint (webhook ban, proxy/firewall, ...)
But what is strange is that it started to fail on 2 different installations, and that you still receive the "connection/disconnection" event. "Normally", either it works at 100% or it fails totally. I'll discuss it with the teams but unfortunately for now I don't see an explanation for this
Concerning the floodlight, the ON/OFF status via a /homestatus request was removed by mistake. I asked the teams to put it back in the JSON response. This being said, its management via the /setstate request still correctly works. So maybe HA considers the device as unreachable because it's waiting for the status ?
Have a good day,
Leslie - Community Manager
Hi @Leslie - and again thanks for staying with us. Together we will solve this 👍
First of all, my understanding is that for whatever the reason could be for banning, there's only one "ban" and that's visible on the dev site, right? Every time I suspect an issue, I check this page, and in the past year or so, I have never once ben banned. Still, the Webhook events are notoriously unreliable and random.
Yesterday, I suddenly had 3 webhook events reported quickly, one after the other, when my wife walked by the camera (as she does all time, every day). I managed to capture the last one, and looks 100% correct. Then, for unknown reasons, the events stopped coming into HA again, but they continue to come into the App. Checked the dev page; No ban.
One question I have; As I have several times tried to delete the Netatmo integration, and set it up again - fresh, but to no avail. However, when setting up the integration, there's an issue with credentials. Check out the last part of the OP entry in this thread; https://community.home-assistant.io/t/how-to-setup-netatmo-integration-using-webhooks-to-speed-up-device-status-update-response-times/385898
It says:
Question: Will the webhook mechanism function if Home Assitant Cloud is selected, instead of the custom App created on the Netatmo Dev page?
Hello Steven,
I performed some tests and indeed I also have huge delays or no notifications at all with the Doorbell webhooks
I asked the teams to investigate and also more information about this event name change and the snapshots disappearence
I warn you once I'll have more information
Have a good day,
Leslie - Community Manager
Hi Leslie,
Thanks for your feedback! We're looking forward to the investigation and hope the issue can be resolved for the 600+ users of our community-developed Homey app for your product :)
The official Athom Homey app for Netatmo also seems to have issues, likely affecting thousands of users experiencing similar problems with the Netamo products intergrate in to Smarthomes with Homey.
Hi Oyvindo,
There was an issue on our side last week leading to some problems with notifications (and, so, impacting webhooks). It seems to have been resolved since Saturday
Do you still have the problem on your side ?
Have a good day,
Leslie - Community Manager
Hi Leslie,
Yes, the problem has gotten worse. I haven't had any webhook events for the past three days.
I checked the dev console today, and the status says OK.
Rgds
Oyvind
By "dev console" you mean the webhook status in your app's details ? (https://dev.netatmo.com/apps/)
Can you try to see in the native Home + Security app if events are correctly sent in your timeline ? If no, I'll report it to the developers
If it works in H+S app you will surely have to redo the /authorize process to see if it fixes the issue
Have a good day,
Leslie - Community Manager
We feel the same. We have one integration with the doorbell that is quite crucial for us, and unfortunately it is very unreliable. Sometimes it does not arrive, most of the times arrives very late.
Last year the webhook was much more stable.
Hello @Leslie, thank you for your reply.
I is it possible to get the request/notification directly from the doorbell (even if on local network) instead of relying on your cloud platform?
The hardware and design of the device is very good, but we are pretty unsatisfied with the service reliability and performance.
If you want more insight on why we need reliable webhooks I've actually published a blog post about our setup.
I've actually tried to deploy the app closer to your infrastructure, but since your outgoing requests come from Microsoft's AS8075 ASN it is hard to figure out where do we get the smallest latency.
Although we truly believe that there's a problem causing delays on your side, if you have any technical recommendation, please do let us know.
@Leslie, Again - thanks for your support. 👍
And yes, I do receive notification events from the H+S App on my Samsung Galaxy S21+ SmartPhone. I must admit I never counted every single one, but my distinct impression is that I get each and everyone according to the settings. I regularly check for webhook ban, and never see any such thing. Proxy/firewall setting hasn't been changed on my end - at least not by me.
I completely agree that it is strange that disconnect events are received (I'm not so sure about connect events). Also, as mentioned - I have two separate installations of HA, both with the Netatmo integration. But for reasons not totally clear to me, both integrations lists all devices and entities from the other site, as well as its own. So it' was difficult for me to determine 100% which site actually receieves the webhook event. What I did then was to make two automations - one in each site, that triggers on any/every Netatmo webhook event and then sends a spoken message to my phone using tts.speak. The message then includes the name of the site so that I can clearly distinguish them.
So far, I haven't been able to extract any pattern - until yesterday - I lost power completely in one site, so now that site i dead until I can get over there and fix it. The power loss had nothing to do with Netatmo or HA. It was a blown fuse due to a short circuit in another part of the house.😶
Hi Leslie,
Just an update.
After a long, silent periode with no Netatmo Events at all, yesterday it suddenly went very chit-chatty and a several Netatmo Events came successively as I stood in front of the camera doing something else. It must have been 5-8 consecutive messages, and then it stopped again. Moving myself away from or into the camera view didn't result in any additional new events being generated.
I have no conclusive pattern to this behavior.
Hi Oyvind,
A fix was pushed nearly 10 days ago and seems to have fixed the delay problems
For this, it seems to be something else for me. I asked the customer support if they have seen similar usecases. I'm waiting for their answer
Have a good day,
Leslie - Community Manager
Hi Luca,
Could you please first confirm that you correctly receive the events via the native Netatmo app ? If it correctly works, it means that the problem stands in the link between your devices and HA
According to the devs we don't have any known issue on our side with webhooks
Did you already tried on your side to undo/redo the webhook access process in HA ?
Have a good day,
Leslie - Community Manager
Allow me to comment ;
We do know that Netatmo's responsibility is between their platform and the App and that Netatmo has no responsibility for the HA integration. Adding to that, we sincerely do appreciate your support Leslie. It isn't obvious to us as users where the problem is located, so it is natural to turn to Netatmo - especialy since the Webhook worked well for years, and then suddenly stopped without anything changed except ordinary HA updates. There has been no new releases of the Netatmo Integration.
In addition, from my side, the webhook events suddenly arrives, randomly, sometimes for a while, then dissapears again for a long time (days, sometimes weeks). When the events arrive, they are perfectly correct (I have captured them and I can send you copies).
The app produces notifications 100% steadily. All the time.
If there is nothing wrong with the Netatmo API, then it should be possible to find other users who can confirm that their integraion works OK, right? Who can help us do that?
If nobody else can found to confirm that webhooks work, then at least we do have a very clear case. If this problem only appears with some of us, then we need to begin debugging to find the common denominator. To do that, we need help and guidance.
Hi Oyvindo,
Indeed, it's difficult to see where is the problem when it deals with 3rd party webhooks
For a reminder, the webhooks sending process is the same when it deals with our native apps or with 3rd party apps (no priority given, no limitations, ...). That's why I always ask for the behavior in the native apps to exclude a potential problem on our end (which is not impossible of course)
Other reminder, we have some cases leading to a temporary webhook ban :
I think that we currently have several hundreds of users on HA now, but I don't see massive complaints about the integration (but maybe the official HA forums have more feedbacks ?)
I see that many HA users are using nabu.casa endpoints for their webhook. I don't know if it's your case but maybe a common denominator could be found here ? (example : slow down on some of their servers where your endpoint is hosted ?)
Have a good day,
Leslie - Community Manager
Hi @Leslie
Could you please first confirm that you correctly receive the events via the native Netatmo app ?
yes
Did you already tried on your side to undo/redo the webhook access process in HA ?
Yes
i'm agree with @oyvindo and thank @leslie for support, what is strange i that sometimes...i receive an event on HA but only sometime....I use Nabucasa
this behavoiur seems to confirm a properly configuration of webhooks on HA otherwise but seems also to goes in case
How can i verify this?
Are you able to verify this on side?
i would like to solve this issue because it's very important for me
Hi Leslie and oyvindo... Today I received 3 events for cams movement, so this mean that the communication is working and the configuration of webhook on HA is correct.
After these.... Not other events...
@luca - that is exactly what I experienced as well: 3 events, and then no more.
Hello,
@luca, we only have a global overview on 3rd party webhooks (for example : the number of banned endpoints). We don't have any info on individual endpoints as they belong to you. Maybe you can use monitoring tools integrated to HA ?
But as oyvindo said, if it was a webhook ban you could see it by yourself in your app's details in the developer portal. Apparently the issue is not linked to a ban. Also, you should not receive any notification at all in this case
@oyvindo, concerning your question about the step 7 of the tutorial ... it's a good question :D
As far as I understand, the "name of credentials" is a kind of variable where needed information to access the developer app are stored (client_secret/secret and so on). I don't know which mechanism stands behind the HA Cloud (authentication, token renewal process, ...), so unfortunately I can't tell you if it could work
All these things being said, I still have the intuition that the problem is linked to the endpoints themselves as the webhooks are partially received on them but always correct via the native Netatmo apps. Could it be possible to put in place some monitoring on them (for example by regularly sending requests to them in order to see if they respond correctly or fail/have a long response time ?)
Have a good day,
Leslie - Community Manager
Hi Leslie,
In order to put in place monitoring on the endpoint in HA, I will need to define a sensor using an HTTP request, i.e. something like this: (do you agree?)
curl -s -o /dev/null -w '%{http_code}' https://<MY_EXTERNAL_URL>/api/webhook/<WEBHOOK_ID>Hi oyvindo,
I'm not developer so I can't help on that :D. But the global idea is to put in place something to regularly poll the endpoint and check if it receives other responses than a normal 200 OK, and if the response delays are correct
Have a good day,
Leslie - Community Manager
Well, that's ok Leslie. I'm confident it would work if I could only find the WEBHOOK_ID
Can you help me on that? (Telling me where to find it. I've searched everywhere!)
Hello,
The problems in this tread are mainly focused on the API intergration for HA. But personally I am also involved in the development of the Homey third party app which also uses the webhooks. Now there are many users complaining that the doorbell trigger no longer works.
Upon investigation, the dev has found that the trigger in the API has been changed from: “NDB-incoming_call” to: “NDB-rtc”
This without any communication, where we also see that the trigger “NDB-rtc” often does not work and no picture is sent.
Is this further known and how can the problem be solved the doorbell intergration in the API is made working again?
Thanks in advance for the response!
Hello Leslie,
Thank you for the feedback and for confirmation, hopefully your colleagues can quickly investigate the problem and apply a solution.
Hope to hear from you soon.
Greets,
Steven
Quickly Steve?? This discussion has been going on for three months now!
Thursday I suddenly had 4 webhook events from my living room camera when I walked by back and forth. Then it silenced again. Friday, I had again several webhook events, from the camera in my study. I managed to capture them all. Then everything went silent again.
It remains a mystery why this webhook thing is so unpredictable.
I want to poll the endpoint, but in order to do that I need the WEBHOOK_ID which I hope Leslie will help me find. (Over the course of this thread, I've learned to be patient and grateful) 😊
Hey, mayby not the right place but can`t find anyweare else to post it.
i have a weather station with all the extra devices, indoor and outdoor cameras.
on the netatmo app everithing is working ( with some delay) but on my Homey nothing is being updated any more ( last update with values is months ago)
I do need the temperature modules to keep my sommerhouse from freezing as i start heating as they become under a certain value.
Is there something that i can try ( so far i tryed deleting and reconnect all devices )
regards
Filip
Please sign in to leave a comment.