Page MenuHomePhabricator

Failed to refresh WME API access token
Closed, ResolvedPublic

Description

message
failed to refresh access token for WME API access
trace
2023-04-21 08:30:17,770 - __main__ - ERROR - ['Traceback (most recent call last):\n', '  File "/usr/local/bin/wm_enterprise_downloader.py", line 144, in refresh\n    \'refresh_token\': self.creds[\'refresh\']},\n', "KeyError: 'refresh'\n"]
2023-04-21 08:30:17,770 - __main__ - ERROR - failed to refresh access token for WME API access
2023-04-21 08:30:17,770 - __main__ - ERROR - ['Traceback (most recent call last):\n', '  File "/usr/local/bin/wm_enterprise_downloader.py", line 442, in get_dump_info\n    self.auth_mgr.refresh()\n', '  File "/usr/local/bin/wm_enterprise_downloader.py", line 144, in refresh\n    \'refresh_token\': self.creds[\'refresh\']},\n', "KeyError: 'refresh'\n"]
2023-04-21 08:30:17,771 - __main__ - ERROR - Failed to retrieve dump file info for wiki miwiktionary and namespace 14, got: b'{"status":401,"message":"token is expired by 15s"}\n'

Event Timeline

Hokwelum created this task.

After Hannah and I poked around a little, it looks like the script now takes more than 24 hours to run, and the auth token we get expires at 24 hours. It may well be that the refresh code has never been tested.

I no longer know who to ping from WME so @RBrounley_WMF you get the honors. Who might be able to let us know about new namespaces being dumped, or planned expansions? Also, it would be lovely to have a test endpoint that gives us an auth token that's good for, say, 5 minutes, for testing purposes; if this is an easy thing, who might we ask about that? Thanks!

We might think about whether to have two downloaders, but isn't AWS supposed to be faster than that?

In the meantime, Hannah will have a patch soon to fix up the method from GET to POST (I wonder if it was GET back in the day and has been changed or if the code was always wrong).

Added @Protsack.stephan and @HShaikh who are more qualified to figure out what's going on here :).

Change 911897 had a related patch set uploaded (by Hokwelum; author: Hokwelum):

[operations/puppet@production] WME refresh token api uses a post request

https://gerrit.wikimedia.org/r/911897

So, to answer the questions point by point:

  • I or someone else on our team can ping in advance to notify about new namespaces/expansions. What's the best way to do that? I had created https://phabricator.wikimedia.org/T331976 to try to give heads up and do some testing in advance on this deployment, but looks like it went under the radar.
  • About the test token endpoint, that's not an easy thing to do due to underlying software that we use for IdM. And the POST request should do the job I think.
  • It depends, usually AWS is fast but sometimes there are bottlenecks in between AWS and the client so:
    • One thing that can be done to make things faster is to download chunks in parallel using Range header. You can first HEAD the endpoint you are going to hit for download, and then using Content-Length you can split the download into chunks using the header mentioned above.

Let me know if there are any more questions.

Tested with the test script https://gist.github.com/apergos/a8239f33226369b8c1e297bb0ad11c78 and the downloader script both living in the dumpsgen user home dir on clouddumps1001, a 5 minute sleep and then refresh, with the latest version of the patch, works great.

Change 911897 merged by ArielGlenn:

[operations/puppet@production] WME refresh token api uses a post request

https://gerrit.wikimedia.org/r/911897

This patch is now deployed. Let's see what happens with the download on the first of the month.

ArielGlenn claimed this task.

Downloads seem to have worked out ok; at least, we do not see errors with refreshing the access token now.