API invalid grant at random intervals

david

Twice today, around 10:00 UTC/12:00 European time and again just now, around 16:00 UTC/18:00 European time, our automated weather downloader got a 401/Bad Request ("invalid grant:") from the Netatmo API. This required us to go into the Netatmo app definition (https://dev.netatmo.com/apps/...) and generate a new token.

Fortunately we store these in a secret provider separate from our app code, but because we're based in Chicago, the 10:00 UTC outage wasn't discovered for 3 hours.

Is there an issue with the API today?

1

Comments

62 comments

Date Votes
  • Comment author
    m.zielinski

    i have the same problem .  did generating a new token help ?

    0
  • Comment author
    david

    @m.zielinski - yes, but we had to do it twice today, which has only happened when we're testing our own code. Updating the token by hand is also a PITA.

    0
  • Comment author
    alexander

    I'm also seeing this issue in my own app as well as in the iPhones apps myatmo and Netatmo Comfort when I log in with the same Netatmo account (email address) on multiple devices.

    The "invalid_grant" error is not returned immediately after logging in on a second device, but the login becomes invalid after a while and refreshing the token fails.

    I have seen this error first at 16:33 (CEST) today (June 3rd, 2024), so I believe Netatmo rolled out a change today that is causing these problems.

    0
  • Comment author
    david

    And it just happened again, around 19:30 UTC/21:30 CEST

    0
  • Comment author
    david

    Each time I refresh the token manually, it asks me to reauthorize the app. This, also, is unusual.

    0
  • Comment author
    alexander

    My app could now successfully refresh the auth token on several device in parallel, so maybe this was a temporary problem.

    0
  • Comment author
    sk

    I am seeing the same problem, refreshing tokens fails, a new authorization is required, but that only helps for a while before refreshing tokens fails again. The refreshToken submitted during request for a new one is OK and valid.

    Can netatmo comment on this, please?

    0
  • Comment author
    sk

    OK, I debugged this several hours now. What I have found out is: refreshing tokens fails after a specific time with an error saying that the access_token is invalid, and that invalidates the grant.

    Re-authorization is required after that to get a fresh grant, however, after a while it starts all over again: invalid access_token >>> invalid grant.

    refresh_token is valid and not changed / updated by some other code.

    Netatmo, please help or comment.

    0
  • Comment author
    Leslie Community moderator
    • Edited

    Hello,

    We just did a modification on the token retrieval process :

    When you refreshed an access_token using the associated endpoint https://api.netatmo.com/oauth2/token, Netatmo servers responded with a couple of tokens : an access_token and a refresh_token. If the previous access_token was still valid, the refresh_token value was never renewed

    Starting from yesterday, this behavior changed to be compliant with the recommendations of the RFC of the OAuth2 Authorization Framework (section 10.4) and improving the security of the data of our users

    When refreshing tokens, access_token and refresh_token values will be automatically renewed and former tokens invalidated

    So, if you do not update and use the new refresh_token value when refreshing your access_token, your users will be disconnected after 3 hours and you will retrieve an “invalid_grant” error

    To fix it, you need to update the tokens value as soon as you get the newly generated ones. If the process is correctly followed, you don't need anymore to redo the /authorize process at each tokens change

    Have a good day,

    Leslie - Community Manager

    1
  • Comment author
    sk

    I am aware of those changes, but: I get an invalid grant error about 10 minutes after refreshing access_token and refresh_token. The refreshed tokens do not prevent getting an invalid grant error, actually an invalid access_token error comes up 10 minutes after refreshing the tokens.

    0
  • Comment author
    sk

    It looks like the tokens get invalidated by the server even if they have not expired yet and even if no invalid tokens have been used to authenticate. And that happens about 10-15 minutes after refreshing tokens.

    0
  • Comment author
    Leslie Community moderator

    @sk, the access_token has a 3 hours lifespan. The refresh_token doesn't have a lifespan so it can only expire in the case a new /token request is sent

    Can you check in your code that you don't have a routine to launch a /token request every 10/15 minutes, making the access/refresh_token pair to be invalidated ?

    Have a good day,

    Leslie - Community Manager

     

    0
  • Comment author
    sk
    • Edited

    @leslie

    My understanding is: if I request new tokens even if the old ones have not expired yet, new ones will simply replace the old ones, and if I then use only the new ones, everything will be fine. Isn't that correct?

    Questions:

    1.) Do I have to wait for existing tokens to expire before requesting fresh ones?

    2.) Will requesting new ones before existing tokens have expired, invalidate the grant?

    Currently my code only requests new tokens if the old one has expired. However, I cannot find any information about the fact that generating new ones before the old one has expired might cause any trouble ...

    Anyway, my access_token still gets invalidated even before it has expired and before refreshing it.

    0
  • Comment author
    sk

    I ran some more tests and found that the access_token always expires about 10 to 20 minutes after refresh or generation, not 3 hours. During that period no token refresh has been initiated by my code, actually the code was sleeping in background without any interaction at all. Once it comes to foreground after about 10 to 20 minutes, the access_token is reported as invalid.

    There must be a reason for that misbehavior on the server side.

    Additionally I sometimes get a 500 HTTP error when trying to access the authentication website once the access_token has expired in order to get a fresh grant.

    Helpless, clueless and frustrated. It really sucks that there's no way to debug that since we have no insight what exactly causes the access_token to get invalidated.

    1
  • Comment author
    Leslie Community moderator

    @sk,

    "if I request new tokens even if the old ones have not expired yet, new ones will simply replace the old ones, and if I then use only the new ones, everything will be fine. Isn't that correct?" <= it's indeed the wanted behavior

    1.) Do I have to wait for existing tokens to expire before requesting fresh ones? <= not necessarily, you can launch your request before it and use new generated values

    2.) Will requesting new ones before existing tokens have expired, invalidate the grant? <= yes, as soon as you receive new values, former ones are immediately invalidated

    I'll ask the question to the devs if they have an idea about your problem. I confirm the access_token expiration is 10800 seconds (otherwise I'd receive hundreds of messages surely). The problem can't come from here

    Can you send me your code via the contact form + the email address of your Netatmo account ? 

    Have a good day,

    Leslie - Community Manager

    0
  • Comment author
    sk

    Have sent you a message. I doubt I am the only one with this issue, see messages above by 'David': "Each time I refresh the token manually, it asks me to reauthorize the app. This, also, is unusual."

    1
  • Comment author
    sk

    I carried out further tests and found that if an app is running on two devices, the access_token on one device invalidates that of the other device. Since both devices are not necessarily connected to each other, there is no possibility to exchange access_token and refresh_token between the devices. This means that authentication is only ever valid on one device and expires as soon as authentication for the same netatmo account is also carried out on another device.

    What solution do you suggest for this problem? It is not possible to exchange the access_token and refresh_token between the devices ...

    0
  • Comment author
    david

    I believe we're doing everything as required. Here is the relevant code (C# 8.0; some logging and defensive code removed):

    public async Task<ApiResponse?> Download(string stationId)
    {
      var attempts = 2;
      try
      {
        HttpResponseMessage httpResponse;
        do
        {
          attempts--;
          httpResponse = await GetStationData(stationId);

          if (httpResponse.StatusCode == HttpStatusCode.OK) break;
          if (httpResponse.StatusCode == HttpStatusCode.Forbidden)
          {
            await RefreshToken();
          }
          else
          {
            return null;
          }
        } while (attempts > 0);
        return await Deserialize<ApiResponse?>(httpResponse);
      }
      catch (Exception ex)
      {
        _logger?.LogError(ex,
          "Exception thrown trying to download Netatmo data for station \"{id}\": {error}",
          stationId, ex.Message);
        return null;
      }
    }

    private async Task<HttpResponseMessage> GetStationData(string stationId)
    {
      var urlText = _settings.EndpointUrl;
      var accessToken = _settings.AccessToken;
      var httpClient = new HttpClient();
      httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);

      return await httpClient.GetAsync(urlText);
    }

    private async Task RefreshToken()
    {
      try
      {
        var httpResponse = await DownloadRefreshToken();
        if (!httpResponse.IsSuccessStatusCode)
        {
          await LogRefreshFailedResponse(httpResponse);
          return;
        }
        await UpdateAccessToken(httpResponse);
      }
      catch (Exception ex)
      {
        _logger?.LogError(ex, "Exception thrown trying to refresh access token: {error}", ex.Message);
      }
    }

    private async Task<HttpResponseMessage> DownloadRefreshToken()
    {
      var expiredToken = _settings.AccessToken;
      using var httpClient = new HttpClient();
      httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", expiredToken);
      var urlText = _settings.TokenUrl;
      var parameters = RefreshTokenParameters;
      return await httpClient.PostAsync(urlText, parameters);
    }

    private HttpContent RefreshTokenParameters
    {
      get
      {
        var clientId = _settings.ClientId;
        var clientSecret = _settings.ClientSecret;
        var refreshToken = _settings.RefreshToken;

        var parameters = new Dictionary<string, string>
        {
          { "client_id", clientId },
          { "client_secret", clientSecret },
          { "grant_type", TokenRequest.RefreshGrantType },
          { "refresh_token", refreshToken }
        };
        return new FormUrlEncodedContent(parameters);
      }
    }

    I mean, what else can we do?

    0
  • Comment author
    sk

    @David: can you confirm that the access_token expires as soon as you run your programme with an authorisation for one and the same netatmo account on more than one device? My guess is: not only within one instance of an app on one device it is necessary to exchange access_token and refresh_token, but also between several, which will be more or less impossible.

    I have tested the whole thing like this:

    1) App authorised on device 1, access_token is valid and remains valid.
    2) App authorised on device 2: this causes the access_token on device 1 to expire

    The whole thing goes on like this, back and forth. Authorisation on device 1 causes the authorisation on device 2 to expire and vice versa. So you can only run an app on one device at a time.

    0
  • Comment author
    sk

    I think this is the cause of the problem: the authorisation is only ever valid on one device and expires as soon as a second device uses the same app with the same netatmo account. So here we now have a real problem, because exchanging the tokens between devices is practically neither possible nor sensible for security reasons.

    Perhaps I have simply overlooked something: in that case, I would be very grateful to know how authentication is supposed to work on two independent devices without de-authenticating one of them.

    0
  • Comment author
    david

    @sk: There is only one device using the API at a time. We actually suspend our production process when we need to test the API directly in our dev environment.

    Using the mobile app or the web dashboard does not seem to affect our automated process. In fact, we can find no correlation between the 401/invalid token response and anything we're doing. It's maddening. The process runs in an Azure function app, so it doesn't carry any state between executions (which are 10 minutes apart).

    Let's see, I renewed the token at 12:00 UTC, and it's now 15:15 UTC...and so far, it's going fine. It conked out last night at 2 am Chicago time so no one caught it for 5 hours, though, which is not fine.

    0
  • Comment author
    david
    • Edited

    Quel drôle et quelle surprise. As soon as I posted that it had run for 5 hours without expiring, it promptly expired. Grace a vous, Netatmo.

    0
  • Comment author
    sk

    Yes, it's maddening, that is for sure. I spent 10 hours today on debugging this, no solution so far.

    0
  • Comment author
    david

    Here are the times that the Netatmo API invalidated our app's token (all times UTC):

    2024-06-03 10:00
    2024-06-03 19:20
    2024-06-03 23:00
    2024-06-04 06:20
    2024-06-04 15:10
    2024-06-04 18:40

    So, starting at noon CEST yesterday, then about every 3 1/2 hours. (There are gaps because, for example, 06:20 UTC is 01:20 in Chicago, and no one was awake to reset the token.)

    0
  • Comment author
    alexander

    @david: in your "UpdateAccessToken" method, are you saving both the new access token AND the new refresh token returned by the server?

    With yesterday's change, the server returns a new refresh token on every refresh, so this must be saved and used when doing the next refresh.

    0
  • Comment author
    david

    @alexander: Yes, because that has been the documented behavior for quite a while.

    private async Task UpdateAccessToken(HttpResponseMessage httpResponse)
    {
      var newTokenResponse = await Deserialize<TokenResponse?>(httpResponse);
      if (newTokenResponse is null)
      {
        _logger?.LogWarning("Token refresh response could not be deserialized");
        return;
      }

      if (string.IsNullOrWhiteSpace(newTokenResponse.AccessToken))
      {
        _logger?.LogWarning("New access token was not returned from API");
        return;
      }

      if (string.IsNullOrWhiteSpace(newTokenResponse.RefreshToken))
      {
        _logger?.LogWarning("New refresh token was not returned from API");
        return;
      }

      _settings.AccessToken = newTokenResponse.AccessToken;
      _settings.RefreshToken = newTokenResponse.RefreshToken;
    }

    Note that we log the condition when an access token is not returned from the API. I haven't seen those warnings in the log. Our logging rules send out an email alert on Warning messages. 

    The warnings we're actually receiving come from the RefreshToken() method (shown a few posts above), when the act of refreshing the token returns 401/token invalid.

    0
  • Comment author
    sk

    I am also doing it like that and it works fine on one device, but once another device requests authorization, the first one gets unauthorized and the access_token is invalidated. And the other way around. So, for me only one device can be authorized at the same time or the access_token gets invalidated.

    0
  • Comment author
    david

    So, re-reading the documentation just now, I have a hypothesis about how the Netatmo API changed.

    When we refresh the token, we apply the expired token in the authorization header. But looking at https://dev.netatmo.com/apidocumentation/oauth#refreshing-a-token, I'm wondering if the refresh POST should just be unauthenticated?

    Netatmo, can you comment on this? Did you change the behavior of https://api.netatmo.com/oauth2/token to fail when it has an expired token in the auth header?

    0
  • Comment author
    david

    The wrench in that machine is that we're receiving invalid_grant, not access_denied or anything indicating the token itself is the problem. Also, when refreshing the token manually, I have been asked to re-authorize the app each time, which I'm not sure is the way it behaved in the past.

    0
  • Comment author
    sk

    Also, when refreshing the token manually, I have been asked to re-authorize the app each time, which I'm not sure is the way it behaved in the past.

    I am also seeing this behavior. Once you try to refresh a token, no grant exists if the token already expired, and because of that a new authorization is required.

    0

Please sign in to leave a comment.