Page MenuHomePhabricator

Repeat OAuth failures in translatewiki.net with Pywikibot
Open, Needs TriagePublic

Description

I'm unable to keep operating my bot in translatewiki.net since aprox. November, 5 due to repeat "Nonce already used" errors.

I'm using Pywikibot stable (3a5006da82c4) on Toolforge.

Example error while using pwb login -verbose -debug (although I get the same error with pwb version)

Python 3.13.5 (main, Jun 25 2025, 18:55:22) [GCC 14.2.0]
Found 3 i18n:i18n processes running, including this one.
WARNING: API error mwoauth-invalid-authorization: The authorization headers in your request are not valid: Nonce already used: 34771845544456520691763062872
           headers=
{'Date': 'Thu, 13 Nov 2025 19:41:13 GMT', 'Content-Type': 'application/json; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Server': 'cloudflare', 'x-content-type-options': 'nosniff', 'mediawiki-api-error': 'mwoauth-invalid-authorization', 'content-security-policy': "default-src 'self'; script-src 'none'; object-src 'none'", 'x-frame-options': 'DENY', 'content-disposition': 'inline; filename=api-result.json', 'Cache-Control': 'private, must-revalidate, max-age=0', 'x-request-id': '0337e3d5b445ed2bcb13da7b', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'Content-Encoding': 'gzip', 'cf-cache-status': 'DYNAMIC', 'vary': 'accept-encoding', 'CF-RAY': '99e0beccbfc58c41-EWR'}
ERROR: Retrying failed OAuth authentication for i18n:i18n: The authorization headers in your request are not valid: Nonce already used: 34771845544456520691763062872

I've created a new OAuth consumer to get new credentials, but it keeps erroring.

Not sure if this is a translatewiki.net issue only, or if this is caused by the OAuth extension or a bug in the Pywikibot code.
Thanks in advance for your assistance.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Is it specific to translatewiki or do you also see it on test.wikipedia.org?

We made a bunch of changes to the OAuth extension around Nov 3 as part of T399199: Update OAuth 2.0 sessions to include new JWT session data from core. I don't see how those would be connected to nonces though, unless this is a secondary error because some earlier request failing made PWB do something incorrect.

Is it specific to translatewiki or do you also see it on test.wikipedia.org?

Hi @Tgr. I don't see the issues I reported on test.wikipedia.org using my Wikimedia bot through Toolforge. Thanks.

Trying again today on translatewiki.net, it keeps erroring:

(pwbvenv) tools.twn-archiver@shell-1764096848:~$ pwb version
Pywikibot: pywikibot/__init__.py (, -1 (unknown), 2025/11/10, 18:51:32, UNKNOWN)
Release version: 10.7.0
packaging version: 25.0
mwparserfromhell version: 0.7.2
wikitextparser version: n/a
requests version: 2.32.5
  cacerts: /data/project/twn-archiver/pwbvenv/lib/python3.13/site-packages/certifi/cacert.pem
    certificate test: ok
Python: 3.13.5 (main, Jun 25 2025, 18:55:22) [GCC 14.2.0]
WARNING: API error mwoauth-invalid-authorization: The authorization headers in your request are not valid: Nonce already used: 180480816234054162811764096866
ERROR: Retrying failed OAuth authentication for i18n:i18n: The authorization headers in your request are not valid: Nonce already used: 180480816234054162811764096866

Is there something I can debug on twn side?

I guess a good first step would be to capture the request ID in pywikibot, and check the relevant server logs.

The nonce store is determined by $wgMWOAuthNonceCacheType, but a store failing in the direction of a lookup unexpectedly succeeding seems unlikely.

Actually not. The relevant check is

if ( !$this->nonceCache->add( $key, 1, $timestamp + 300 ) ) {

which cannot differentiate between the key already being used and other kinds of store error, so maybe whatever BagOStuff backend TWN is using for this is unreliable.

Oh, right, we had a task about this (which got closed but not really fixed): T106066: Don't show "Nonce already used" error on memcache failure

Hi. This is happening again for my ArchiveBot account at translatewiki.net. I cleared all cron jobs for the bot yesterday and I've tried right now to re-run it again without success:

WARNING: API error mwoauth-invalid-authorization: The authorization headers in your request are not valid: Nonce already used: 109481264737078862581783002661
ERROR: Retrying failed OAuth authentication for i18n:i18n: The authorization headers in your request are not valid: Nonce already used: 109481264737078862581783002661
WARNING: API error mwoauth-invalid-authorization: The authorization headers in your request are not valid: Nonce already used: 67118816203591375631783002661
ERROR: Retrying failed OAuth authentication for i18n:i18n: The authorization headers in your request are not valid: Nonce already used: 67118816203591375631783002661

Please let me know if you need further information, data or action from me. Thank you.

ArchiveBot was making a huge amount of requests so I blocked it briefly on June 30th. I have restarted memcached in case that helps.

ArchiveBot was making a huge amount of requests so I blocked it briefly on June 30th.

@Nikerabbit I'm sorry, that shouldn't be happening. The bot only runs an archiving script once per day. Maybe something got stuck in toolforge-jobs/k8s?

I have restarted memcached in case that helps.

Thanks. I've tried again, but it's failing again with the same error:

WARNING: API error mwoauth-invalid-authorization: The authorization headers in your request are not valid: Nonce already used: 831803374834379871783003815
ERROR: Retrying failed OAuth authentication for i18n:i18n: The authorization headers in your request are not valid: Nonce already used: 831803374834379871783003815
WARNING: API error mwoauth-invalid-authorization: The authorization headers in your request are not valid: Nonce already used: 128042723829343312721783003815
ERROR: Retrying failed OAuth authentication for i18n:i18n: The authorization headers in your request are not valid: Nonce already used: 128042723829343312721783003815
WARNING: API error mwoauth-invalid-authorization: The authorization headers in your request are not valid: Nonce already used: 89985283607854261991783003815
ERROR: Retrying failed OAuth authentication for i18n:i18n: The authorization headers in your request are not valid: Nonce already used: 89985283607854261991783003815

The other issue I can think of that would produce nonce errors (besides memcached being unreliable) is T429666: When using local ids, OAuth double checks nonce resulting in every request failing. Check the OAuth log channel, there should be a stack trace for the nonce check.

I'm sorry but I'm still unable to run the bot on-wiki, not even non-write scripts:

(pwbvenv) tools.twn-archiver@shell-1783263755:~$ pwb version -v -debug
Pywikibot: pywikibot/__init__.py (, -1 (unknown), 2026/07/01, 19:11:33, UNKNOWN)
Release version: 11.5.0.dev2
packaging version: 26.2
mwparserfromhell version: 0.7.2
wikitextparser version: n/a
requests version: 2.34.2
  cacerts: /data/project/twn-archiver/pwbvenv/lib/python3.13/site-packages/certifi/cacert.pem
Python: 3.13.5 (main, Jun 25 2025, 18:55:22) [GCC 14.2.0]
WARNING: API error mwoauth-invalid-authorization: The authorization headers in your request are not valid: Nonce already used: 122015241732693265801783263944
ERROR: Retrying failed OAuth authentication for i18n:i18n: The authorization headers in your request are not valid: Nonce already used: 122015241732693265801783263944
WARNING: API error mwoauth-invalid-authorization: The authorization headers in your request are not valid: Nonce already used: 116321052004399857681783263944
ERROR: Retrying failed OAuth authentication for i18n:i18n: The authorization headers in your request are not valid: Nonce already used: 116321052004399857681783263944
WARNING: API error mwoauth-invalid-authorization: The authorization headers in your request are not valid: Nonce already used: 12941063261542145921783263944
ERROR: Retrying failed OAuth authentication for i18n:i18n: The authorization headers in your request are not valid: Nonce already used: 12941063261542145921783263944

If it helps, I'm providing below detailed output from login.py:

Python 3.13.5 (main, Jun 25 2025, 18:55:22) [GCC 14.2.0]
Found 1 i18n:i18n processes running, including this one.
Sleeping for 0.1 seconds, 2026-07-05 15:17:19
WARNING: API error mwoauth-invalid-authorization: The authorization headers in your request are not valid: Nonce already used: 7029307832526492541783264639
           headers=
{'Date': 'Sun, 05 Jul 2026 15:17:20 GMT', 'Content-Type': 'application/json; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Server': 'cloudflare', 'X-Content-Type-Options': 'nosniff', 'Mediawiki-Api-Error': 'mwoauth-invalid-authorization', 'Content-Security-Policy': "default-src 'self'; script-src 'none'; object-src 'none'", 'X-Frame-Options': 'DENY', 'Content-Disposition': 'inline; filename=api-result.json', 'Cache-Control': 'private, must-revalidate, max-age=0', 'X-Request-Id': '755937394ff71364ca14c0bc', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'Content-Encoding': 'gzip', 'Cf-Cache-Status': 'DYNAMIC', 'Vary': 'accept-encoding', 'CF-RAY': 'a16755fdde241084-ORD', 'alt-svc': 'h3=":443"; ma=86400'}
ERROR: Retrying failed OAuth authentication for i18n:i18n: The authorization headers in your request are not valid: Nonce already used: 7029307832526492541783264639

I'm seeing the same 'CF-RAY': 'a167685b7a58f606-ORD'. Maybe translatewiki CloudFlare is making the request to be replayed or delivered twice, thus making MediaWiki think the nonce is being reused?

I'm seeing the same 'CF-RAY': 'a167685b7a58f606-ORD'. Maybe translatewiki CloudFlare is making the request to be replayed or delivered twice, thus making MediaWiki think the nonce is being reused?

That was suspected last time this happened as well, but I found no evidence of that.

Are you saying you are seeing the same CF-RAY ID multiple times? In theory Cloudflare doesn't do that, not even for cached requests, and these are not cached (IIUIC Cf-Cache-Status: DYNAMIC is the equivalent of X-Cache-Status: pass in Varnish). So that (along with X-Request-Id also being identical) would imply some unexpected caching happening downstream of Cloudflare.

(Which in theory should be prevented by Cache-Control: private, must-revalidate, max-age=0. Although I don't know why we don't just set Cache-Control: no-store on OAuth handshakes.)

Are you saying you are seeing the same CF-RAY ID multiple times?

@Tgr. Sorry. I should've looked closely. The font size is small and didn't notice that the CF-RAY ID, although similar, is not identical. I pasted at P94734 the output of a pwb login -verbose -debug I've just attempted just in case it's helpful.

FWIW yesterday, I did switch the bot's login method from OAuth to BotPasswords, so I could run some tasks. Not sure where the "lock" is, but as you can see in the linked paste, MediaWiki is still refusing to allow me to login via OAuth even after several days have already passed.

Perhaps Pywikibot should not endlessly retry to avoid T410085#12081246?