Page MenuHomePhabricator
Create Task
Maniphest T347223

Exception: Key contains invalid characters: centralauth:central-login-complete-token:1�À§À¢%2527%2522
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
labels.normalized_message
[{reqId}] {exception_url}   Exception: Key contains invalid characters: centralauth:central-login-complete-token:1�À§À¢%2527%2522
error.stack_trace
from /srv/mediawiki/php-1.41.0-wmf.27/includes/libs/objectcache/MemcachedBagOStuff.php(109)
#0 /srv/mediawiki/php-1.41.0-wmf.27/includes/libs/objectcache/MemcachedBagOStuff.php(120): MemcachedBagOStuff->validateKeyEncoding(string)
#1 /srv/mediawiki/php-1.41.0-wmf.27/includes/libs/objectcache/MemcachedPeclBagOStuff.php(182): MemcachedBagOStuff->validateKeyAndPrependRoute(string)
#2 /srv/mediawiki/php-1.41.0-wmf.27/includes/libs/objectcache/MediumSpecificBagOStuff.php(117): MemcachedPeclBagOStuff->doGet(string, integer)
#3 /srv/mediawiki/php-1.41.0-wmf.27/extensions/CentralAuth/includes/CentralAuthUtilityService.php(96): MediumSpecificBagOStuff->get(string)
#4 [internal function]: MediaWiki\Extension\CentralAuth\CentralAuthUtilityService::MediaWiki\Extension\CentralAuth\{closure}()
#5 /srv/mediawiki/php-1.41.0-wmf.27/vendor/wikimedia/wait-condition-loop/src/WaitConditionLoop.php(96): call_user_func(Closure)
#6 /srv/mediawiki/php-1.41.0-wmf.27/extensions/CentralAuth/includes/CentralAuthUtilityService.php(107): Wikimedia\WaitConditionLoop->invoke()
#7 /srv/mediawiki/php-1.41.0-wmf.27/extensions/CentralAuth/includes/Special/SpecialCentralLogin.php(241): MediaWiki\Extension\CentralAuth\CentralAuthUtilityService->getKeyValueUponExistence(MemcachedPeclBagOStuff, string)
#8 /srv/mediawiki/php-1.41.0-wmf.27/extensions/CentralAuth/includes/Special/SpecialCentralLogin.php(91): MediaWiki\Extension\CentralAuth\Special\SpecialCentralLogin->doLoginComplete(string)
#9 /srv/mediawiki/php-1.41.0-wmf.27/includes/specialpage/SpecialPage.php(721): MediaWiki\Extension\CentralAuth\Special\SpecialCentralLogin->execute(string)
#10 /srv/mediawiki/php-1.41.0-wmf.27/includes/specialpage/SpecialPageFactory.php(1622): MediaWiki\SpecialPage\SpecialPage->run(string)
#11 /srv/mediawiki/php-1.41.0-wmf.27/includes/MediaWiki.php(354): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
#12 /srv/mediawiki/php-1.41.0-wmf.27/includes/MediaWiki.php(953): MediaWiki->performRequest()
#13 /srv/mediawiki/php-1.41.0-wmf.27/includes/MediaWiki.php(601): MediaWiki->main()
#14 /srv/mediawiki/php-1.41.0-wmf.27/index.php(50): MediaWiki->run()
#15 /srv/mediawiki/php-1.41.0-wmf.27/index.php(46): wfIndexMain()
#16 /srv/mediawiki/w/index.php(3): require(string)
#17 {main}
Impact
Notes

Probably triggered by tempering with URLs. Reproducible with GET request.

Details

Request URL
https://www.mediawiki.org/wiki/Special:CentralLogin/complete?token=*
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Drop "$wgCentralAuthDatabase:$keygroup:$components..." format (p.3)mediawiki/extensions/CentralAuthmaster+43 -145
Try reading from newKey first before falling back to oldKey (p.2)mediawiki/extensions/CentralAuthmaster+38 -36
Fix centralauthtoken key schema migrationmediawiki/extensions/CentralAuthmaster+1 -4
Begin migrating key formats for cache key lookups or writes (p.1)mediawiki/extensions/CentralAuthmaster+181 -48
Fix centralauthtoken key schema migrationmediawiki/extensions/CentralAuthwmf/1.42.0-wmf.4+1 -4
Customize query in gerrit

Related Objects

Event Timeline

Ammarpad created this task.Sep 23 2023, 7:09 AM
Restricted Application added a project: MediaWiki-Platform-Team. · View Herald TranscriptSep 23 2023, 7:09 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Ammarpad updated the task description. (Show Details)Sep 23 2023, 7:17 AM
Krinkle moved this task from Untriaged to Sep 2023 on the Wikimedia-production-error board.Sep 25 2023, 1:17 PM
Bmueller moved this task from Inbox, needs triage to In progress (DO NOT USE) on the MediaWiki-Platform-Team board.Sep 25 2023, 1:31 PM
Bmueller assigned this task to DAlangi_WMF.Sep 25 2023, 1:36 PM
gerritbot added a comment.Sep 25 2023, 3:44 PM

Change 960643 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/extensions/CentralAuth@master] Escape/normalize cache key before retreiving data from cache backend

https://gerrit.wikimedia.org/r/960643

gerritbot added a project: Patch-For-Review.Sep 25 2023, 3:44 PM
DAlangi_WMF changed the task status from Open to In Progress.Sep 28 2023, 4:21 PM
• larissagaulia moved this task from In progress (DO NOT USE) to Waiting on the MediaWiki-Platform-Team board.Oct 9 2023, 1:41 PM
gerritbot added a comment.Oct 9 2023, 2:44 PM

Change 961794 had a related patch set uploaded (by Krinkle; author: Derick Alangi):

[mediawiki/extensions/CentralAuth@master] Drop support for: "$wgCentralAuthDatabase:$keygroup:$components..." key

https://gerrit.wikimedia.org/r/961794

DAlangi_WMF moved this task from Waiting to In progress (DO NOT USE) on the MediaWiki-Platform-Team board.Oct 23 2023, 8:46 AM
gerritbot added a comment.Oct 23 2023, 10:36 AM

Change 967880 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/extensions/CentralAuth@master] Try reading from newKey first before falling back to oldKey (p.2)

https://gerrit.wikimedia.org/r/967880

gerritbot added a comment.Oct 23 2023, 10:45 AM

Change 961794 had a related patch set uploaded (by D3r1ck01; author: Derick Alangi):

[mediawiki/extensions/CentralAuth@master] Drop "$wgCentralAuthDatabase:$keygroup:$components..." format (p.3)

https://gerrit.wikimedia.org/r/961794

gerritbot added a comment.Nov 6 2023, 5:25 AM

Change 960643 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Begin migrating key formats for cache key lookups or writes (p.1)

https://gerrit.wikimedia.org/r/960643

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.4; 2023-11-07).Nov 6 2023, 6:00 AM
Tgr subscribed.Nov 7 2023, 7:03 AM

We should probably test the changes (all three patches) after they are deployed by the train. I think a test run on Tuesday (ie. after new code got deployed to group0) would look something like this:

  • log in on some group 0 wiki (let's say akwiki)
  • check that shared cookies work by checking in on a wiki on the same second-level domain (e.g. howiki)
  • check that central login works and edge login works by checking that you are logged in at another group 0 wiki that is edge login enabled (e.g. mediawikiwiki) and a group 1 wiki that's edge login enabled (e.g. metawiki)
  • check that central autologin works by logging in on some group 0 wiki that is not edge login enabled (e.g. wikimania2016wiki) and some group 1 wiki that's not edge login enabled (e.g. outreachwiki)
  • get a token using action=centralauthtoken in Special:ApiSandbox, use it to authenticate an action API request and a REST API request. (Note the token is only valid for 10 sec.) The REST API doesn't have any endpoint where the authenticated and the unathenticated response would differ, but an incorrect token will result in an error, so it's still testable.

A test on Wednesday would look like this:

  • repeat the steps above (to test new code <-> new code communication)
  • repeat the steps above again, but using group 2 wikis instead of group 0 wikis (to test new code <-> old code communcation). There is no group2 that isn't edge login enabled so that part can be skipped.

There is no need to test on Thursday, the previous two tests should already cover all the possible code pathes with all the possible old code / new code combinations.

(Note that these tests require third-party cookie blocking to be disabled in the browser (default for some browsers, but not for all). There is a guide on how the various authentication flows work, and debugging instructions (which are not very user-friendly and shouldn't be needed, but just in case).)

Tgr mentioned this in T350723: GlobalWatchlist: cross-wiki request to mediawiki.org or test.wikipedia.org failing with `badtoken`.Nov 7 2023, 8:07 PM
gerritbot added a comment.Nov 7 2023, 8:36 PM

Change 972464 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/CentralAuth@master] Fix centralauthtoken key schema migration

https://gerrit.wikimedia.org/r/972464

gerritbot added a comment.Nov 7 2023, 9:00 PM

Change 972464 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Fix centralauthtoken key schema migration

https://gerrit.wikimedia.org/r/972464

gerritbot added a comment.Nov 7 2023, 9:08 PM

Change 972262 had a related patch set uploaded (by Gergő Tisza; author: Gergő Tisza):

[mediawiki/extensions/CentralAuth@wmf/1.42.0-wmf.4] Fix centralauthtoken key schema migration

https://gerrit.wikimedia.org/r/972262

gerritbot added a comment.Nov 7 2023, 9:44 PM

Change 972262 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@wmf/1.42.0-wmf.4] Fix centralauthtoken key schema migration

https://gerrit.wikimedia.org/r/972262

Stashbot added a comment.Nov 7 2023, 9:45 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-07T21:45:03Z] <tgr@deploy2002> Started scap: Backport for [[gerrit:972262|Fix centralauthtoken key schema migration (T347223 T350723)]]

Stashbot added a comment.Nov 7 2023, 9:46 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-07T21:46:23Z] <tgr@deploy2002> tgr: Backport for [[gerrit:972262|Fix centralauthtoken key schema migration (T347223 T350723)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Nov 7 2023, 9:58 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-07T21:58:20Z] <tgr@deploy2002> Finished scap: Backport for [[gerrit:972262|Fix centralauthtoken key schema migration (T347223 T350723)]] (duration: 13m 17s)

• Aboodhassanio subscribed.Nov 7 2023, 10:30 PM
gerritbot added a comment.Nov 12 2023, 11:58 PM

Change 967880 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Try reading from newKey first before falling back to oldKey (p.2)

https://gerrit.wikimedia.org/r/967880

ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.5; 2023-11-14); removed MW-1.42-notes (1.42.0-wmf.4; 2023-11-07).Nov 13 2023, 1:00 AM
gerritbot added a comment.Nov 19 2023, 6:27 AM

Change 961794 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] Drop "$wgCentralAuthDatabase:$keygroup:$components..." format (p.3)

https://gerrit.wikimedia.org/r/961794

ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.7; 2023-11-28); removed MW-1.42-notes (1.42.0-wmf.5; 2023-11-14).Nov 19 2023, 7:00 AM
• larissagaulia moved this task from In progress (DO NOT USE) to Waiting on the MediaWiki-Platform-Team board.Nov 20 2023, 3:41 PM
DAlangi_WMF added a comment.Nov 22 2023, 11:17 AM

(update): The last patch on the chain of patches is riding the train this week. At this point, the migration is already complete and we're dropping support for the old key format. No incidents so far and things are working as expected.

By tomorrow, the aforementioned patch will hit group 2 wikis (by end of day) and if everywhere is still silent, I'll go ahead and resolve this ticket. Also, logstash looks clear but I'll like to hear from @Ammarpad for further confirmation ☺️ on logstash from their side.

Jdforrester-WMF subscribed.Nov 22 2023, 2:31 PM
In T347223#9352116, @DAlangi_WMF wrote:

(update): The last patch on the chain of patches is riding the train this week. At this point, the migration is already complete and we're dropping support for the old key format. No incidents so far and things are working as expected.

By tomorrow, the aforementioned patch will hit group 2 wikis (by end of day) and if everywhere is still silent, I'll go ahead and resolve this ticket. Also, logstash looks clear but I'll like to hear from @Ammarpad for further confirmation ☺️ on logstash from their side.

This will happen next week with wmf.7; there's no train this week.

DAlangi_WMF added a comment.Nov 22 2023, 2:36 PM

This will happen next week with wmf.7; there's no train this week.

Thanks for the indication @Jdforrester-WMF. I missed that entirely. Next week then!

DAlangi_WMF closed this task as Resolved.Nov 30 2023, 3:04 PM
DAlangi_WMF added a subscriber: Krinkle.

All clear, this can be resolved now. Thanks for the collaboration @Tgr, @Krinkle and @Jdforrester-WMF.

Most especially, thanks @Ammarpad for reporting.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits