Risky Patch! 🚂🔥

• Change: not a single patch, rather, a group of risky changes in CentralAuth
  • Example changes: https://gerrit.wikimedia.org/r/664866, https://gerrit.wikimedia.org/r/709850, https://gerrit.wikimedia.org/r/664895
• Summary: (Why is it risky?)
  • Significant refactoring of several classes in the CentralAuth extension (600+ lines changed in total).
  • CentralAuth is a critical extension for the wikis, but unfortunately not covered by tests well.
  • Manual tests were done by multiple developers, but they're (on average) less reliable than automated tests.
• Test plan:
  • Local testing
  • Beta testing
  • Monitoring of CentralAuth breakage in logs and by real users
• Places to monitor:
  • Logstash: mediawiki-errors
  • Grafana: mediawiki-errors
  • User reports
• Revert plan: Rollback train.
• Affected wikis: All wikis running CentralAuth
• IRC Contact: majavah, urbanecm
• UBN Task Projects/tags: MediaWiki-extensions-CentralAuth

CC @Majavah.

For your information, during deploying Growth features to a bunch of wikis (44 wikis in total), I accidentally managed to take zh_yuewiki down. This caused a lot of exceptions at that wiki for a few minutes (until I noticed and fixed the issue).

This is a snippet from -operations around that time:

17:13 <urbanecm> I just accidentaly brought https://zh-yue.wikipedia.org/ down. Fixing...
17:14 <urbanecm> and zh_yuewiki is back online now. Sorry!
17:16 <urbanecm> !log [urbanecm@mwmaint2002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=zh_yuewiki growthexperiments # T289680

If there are any "table does not exist" exceptions at zh_yuewiki from today, that's very likely related.

For your reference, [89f97b03-ad58-4a25-9732-988a33f2862e] /wiki/Special:EditGrowthConfig Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'zh_yuewiki.growthexperiments_mentor_mentee' doesn't exist (db2115) is an example error.

Removed T289692: api.wikimedia.org fatal exception: Error: Call to a member function text() on string as a blocker. This error is not a result of a recent code change, and does not make the train deployment unsafe.

Mentioned in SAL (#wikimedia-operations) [2021-08-25T20:02:52Z] <brennen> 1.37.0-wmf.20 (T281161) status: blocked at group0; 2/3 blockers have probable patches, all seem to be getting attention, so holding off on blocker mail for now.

Mentioned in SAL (#wikimedia-operations) [2021-08-25T21:59:53Z] <brennen> 1.37.0-wmf.20 train status (T281161) blockers should be patched shortly; as we've reached the 15:00 Pacific deploy cutoff for the day, train will resume first thing in US morning

API linkshere/fileusage doesn't report links/uses through redirects anymore: T289792. This may be a blocker. It could result in a lot of broken file links/uses when a file is moved on Commons and file usage isn't reported properly. Also affects non-file pages.

And no, I'm not using double redirects.

@brennen, Jdforrester-WMF seems confused. This will cause problems that won't correct themselves. If this happens on production, you'll have to roll back Mediawiki. I assume rolling back Mediawiki is not fun.

Mentions:
T281932
T285978
T287630
T288151
T289188
T289351
T289625
T289652
T289692
T289697
T289717
T289728
T289730
T289731
T289792
T289806

T289792 is reproducible on test.wikipedia.org so it may belong here after all?
Never mind my bad it isn't I think.

In T281161#7313540, @AlexisJazz wrote:

T289792 is reproducible on test.wikipedia.org so it may belong here after all?

I suggest to discuss further in the task itself to avoid discussion fragmentation.

Change 715773 had a related patch set uploaded (by 20after4; author: 20after4):

[operations/mediawiki-config@master] testwikis wikis to 1.37.0-wmf.21 refs T281161

https://gerrit.wikimedia.org/r/715773

Change 715773 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.37.0-wmf.21 refs T281161

https://gerrit.wikimedia.org/r/715773

Mentioned in SAL (#wikimedia-operations) [2021-08-31T18:30:04Z] <twentyafterfour@deploy1002> Started scap: testwikis wikis to 1.37.0-wmf.21 refs T281161

Mentioned in SAL (#wikimedia-operations) [2021-08-31T19:05:58Z] <twentyafterfour@deploy1002> Finished scap: testwikis wikis to 1.37.0-wmf.21 refs T281161 (duration: 35m 53s)

Change 715785 had a related patch set uploaded (by 20after4; author: 20after4):

[operations/mediawiki-config@master] group0 wikis to 1.37.0-wmf.21 refs T281161

https://gerrit.wikimedia.org/r/715785

Change 715785 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.37.0-wmf.21 refs T281161

https://gerrit.wikimedia.org/r/715785

Mentioned in SAL (#wikimedia-operations) [2021-08-31T20:00:30Z] <twentyafterfour@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.37.0-wmf.21 refs T281161

Change 716024 had a related patch set uploaded (by 20after4; author: 20after4):

[operations/mediawiki-config@master] group1 wikis to 1.37.0-wmf.21 refs T281161

https://gerrit.wikimedia.org/r/716024

Change 716024 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.37.0-wmf.21 refs T281161

https://gerrit.wikimedia.org/r/716024

Mentioned in SAL (#wikimedia-operations) [2021-09-01T19:56:34Z] <twentyafterfour@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.21 refs T281161

Mentioned in SAL (#wikimedia-operations) [2021-09-01T19:57:44Z] <twentyafterfour@deploy1002> Synchronized php: group1 wikis to 1.37.0-wmf.21 refs T281161 (duration: 01m 06s)

sorry for spamming this task with wmf.21 updates, I had the wrong task id somehow