Risky Patch! 🚂🔥

• Change: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/672487
• What it does/What it's risky:
  • Makes substantial changes to the minifier used to generate minified JS in ResourceLoader, to provide ES6 support
  • These changes may introduce bugs. Minification of ES6 code (new syntax) could break because the new features are imperfect, or we could have introduced regressions that break minification of existing ES5 code.
  • Minifier bugs are subtle, difficult to test for or isolate, and generally only arise in the wild. They tend to result in strange and confusing runtime JS errors.
• Test Plan:
  • The unit test suite is extensive, but this is the kind of thing with a long tail of edge cases that are found in the wild
  • Wikibase CI turns out to test some of this already, that's how we found out it broke the first time (T277161). Their CI passes now, which is a hopeful sign.
  • No breakage has been reported in beta labs
  • After the deploying to production, watch for an increase in JS errors, and reports of broken JS features
• Places to Watch for Breakage
  • JS errors dashboard: https://grafana.wikimedia.org/d/000000566/overview?tab=alert&editPanel=16&orgId=1 (a spike in errors here could be caused by this change, or by something else)
• Revert Plan: revert change 672487 and the associated vendor change
• Wikis Affected: all
• IRC Contact: @Catrope (RoanKattouw on IRC)
• UBN Task Projects/tags: Wikimedia-Minify Design-Systems-team-20200324-20220422 MediaWiki-ResourceLoader

Thanks for the risk report @Catrope

Risky Patch! 🚂🔥

• Change: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/FlaggedRevs/+/672570 (in general the whole extension)
• What it does/What it's risky:
  • FlaggedRevs is a big mess and we are trying to unravel some of the code and simplify its logic.
  • FlaggedRevs is not well covered by tests and has a ton of edge cases.
  • It has lots of configuration that differ from wiki to wiki, it can take many shapes and forms.
• Test Plan:
  • FlaggedRevs is enabled on test2wiki. Mostly manual testing unfortunately and log watching.
  • This extension has many options and they differ from wiki to wiki, meaning it can be fine in one but completely broken in another.
• Places to Watch for Breakage
  • Logstash MediaWiki Errors Dashboard: https://logstash.wikimedia.org/app/kibana#/dashboard/mediawiki-errors
  • Grafana Production Logging Dashboard: https://grafana.wikimedia.org/d/000000102/production-logging
• Revert Plan: rollback
• Wikis Affected:

alswiki
arwiki
bewiki
bnwiki
bswiki
cawikinews
cewiki
ckbwiki
dewiki
dewikiquote
dewiktionary
elwikinews
enwiki
enwikibooks
enwikinews
eowiki
eswikinews
fawiki
fawikinews
fiwiki
frwikinews
hewikisource
hiwiki
huwiki
iawiki
idwiki
iswiktionary
kawiki
lawikisource
mkwiki
plwiki
plwikisource
plwiktionary
ptwikibooks
ptwikinews
ptwikisource
ruwiki
ruwikinews
ruwikiquote
ruwikisource
ruwiktionary
sqwiki
tawikinews
test2wiki
trwiki
trwikiquote
ukwiki
ukwiktionary
vecwiki
zh_classicalwiki

• IRC Contact: @Ladsgroup (Amir1)
• UBN Task Projects/tags: FlaggedRevs

Change 674183 had a related patch set uploaded (by Jforrester; owner: trainbranchbot):
[mediawiki/core@wmf/1.36.0-wmf.36] Branch commit for wmf/1.36.0-wmf.36

https://gerrit.wikimedia.org/r/674183

Change 674183 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.36] Branch commit for wmf/1.36.0-wmf.36

https://gerrit.wikimedia.org/r/674183

Mentioned in SAL (#wikimedia-operations) [2021-03-23T09:53:42Z] <hashar> scap prep 1.36.0-wmf.36 # T274940

Mentioned in SAL (#wikimedia-operations) [2021-03-23T10:01:01Z] <hashar> Applied security patches for 1.36.0-wmf.36 # T274940

Mentioned in SAL (#wikimedia-operations) [2021-03-23T10:02:34Z] <hashar> scap clean --delete 1.36.0-wmf.32 # T274940

Mentioned in SAL (#wikimedia-operations) [2021-03-23T13:23:10Z] <hashar@deploy1002> Started scap: Promote testwikis from 1.36.0-wmf.35 to 1.36.0-wmf.36 - T274940

Mentioned in SAL (#wikimedia-operations) [2021-03-23T13:55:05Z] <hashar@deploy1002> Finished scap: Promote testwikis from 1.36.0-wmf.35 to 1.36.0-wmf.36 - T274940 (duration: 31m 57s)

I have synchronized php: group1 wikis to 1.36.0-wmf.36

Change 674662 had a related patch set uploaded (by Hashar; author: Hashar):
[operations/mediawiki-config@master] Revert "group1 wikis to 1.36.0-wmf.36"

https://gerrit.wikimedia.org/r/674662

Change 674662 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "group1 wikis to 1.36.0-wmf.36"

https://gerrit.wikimedia.org/r/674662

A minute on group 1 wikis and we got three issues:

• T278375: Class 'GlobalUsageHooks' not found
• T278376: Parser Use of RevisionRecord for a page that can't exist: (eg Special:MyLanguage/Main Page)
• T278379: Argument 1 passed to ProofreadPage\Index\IndexTemplateStyles::__construct() must be an instance of Title, null given, called in /srv/mediawiki/php-1.36.0-wmf.36/extensions/ProofreadPage/includes/Page/PageContent.php on line 284

I have sent an email to wikitech-l / ops-l mailing list

T278375 got hotfixed

T278376 has been determined as not being a blocker (deprecation warning)

T278379 had the faulty commit reverted

I have pushed 1.36.0-wmf.36 to group 1 at 20:14 UTC and the errors are gone.

@Catrope @Ladsgroup looks like your risky patches went well for group 1! No errors about FlaggedRevs and I can't see any surge of errors coming from client side javascript. group 2 might be a different story, but that is for tomorrow.

I am gone for now :]

Change 674958 had a related patch set uploaded (by Hashar; author: Hashar):
[operations/mediawiki-config@master] Revert "ruwiki: flaggedrevs: Delete autoeditor group"

https://gerrit.wikimedia.org/r/674958

Mentioned in SAL (#wikimedia-operations) [2021-03-25T19:37:54Z] <hashar@deploy1002> rebuilt and synchronized wikiversions files: Revert group2 wikis to 1.36.0-wmf.35 - T274940

Change 674958 abandoned by Hashar:
[operations/mediawiki-config@master] Revert "ruwiki: flaggedrevs: Delete autoeditor group"

Reason:

https://gerrit.wikimedia.org/r/674958

Change 674960 had a related patch set uploaded (by Hashar; author: Hashar):
[operations/mediawiki-config@master] Revert "all wikis to 1.36.0-wmf.36"

https://gerrit.wikimedia.org/r/674960

Change 674961 had a related patch set uploaded (by Hashar; author: Hashar):
[operations/mediawiki-config@master] Revert "group1 wikis to 1.36.0-wmf.36"

https://gerrit.wikimedia.org/r/674961

Mentioned in SAL (#wikimedia-operations) [2021-03-25T19:48:20Z] <hashar@deploy1002> rebuilt and synchronized wikiversions files: Revert group 1 and 2 wikis to 1.36.0-wmf.35 - T274940

Change 674960 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "all wikis to 1.36.0-wmf.36"

https://gerrit.wikimedia.org/r/674960

Change 674961 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "group1 wikis to 1.36.0-wmf.36"

https://gerrit.wikimedia.org/r/674961

Blocker T277795 - should finally be solved by https://gerrit.wikimedia.org/r/c/mediawiki/core/+/674836

I have added T278562 as a subtask since that is related to the roll out of 1.36.0-wmf.36. mediawiki/core has a feature relying on the wiki page MediaWiki:Autoblock_whitelist and it got proposed for renaming. That has been done on commonswiki after .36 got deployed but when we rolled back to .35 the code obviously did not know about the new page name. It is hard :]

In T274940#6949275, @hashar wrote:

I have added T278562 as a subtask since that is related to the roll out of 1.36.0-wmf.36. mediawiki/core has a feature relying on the wiki page MediaWiki:Autoblock_whitelist and it got proposed for renaming. That has been done on commonswiki after .36 got deployed but when we rolled back to .35 the code obviously did not know about the new page name. It is hard :]

T278562 is restricted for me. Maybe the following is not related: I restored the content of the old message MediaWiki:Autoblock_whitelist this morning after I read about an issue: https://commons.wikimedia.org/wiki/Commons:Village_pump#CropTool_returns_%22Blocked_user%22_error_message

@Raymond yes that is what the task is about and you have fixed it :-] I have attached the task to this blocker so that everyone is aware of it.

Meanwhile I have deployed two hotfixes (with the assistance of @Urbanecm):

• T277795: User not found by actor ID: [id]
• T278429: Argument 1 passed to MediaWiki\Extension\Translate\PageTranslation\ParserOutput::sourcePageTextForRendering() must be an instance of Language, instance of StubUserLang given, called in /srv/mediawiki/php-1.36.0-wmf.36/extensions/Translate/tag/PageTranslationHooks.php on line 53

The last blocker would thus be FlaggedRevs / T278478 which will be pushed on Monday morning.

Then I guess I will push 1.36.0-wmf.36 on all wikis on monday at 14:00 CEST (noon UTC).

The fix for flagged revs is merged on master and cherry-picked on wmf.36, if you want to deploy it now, I'm around. If not, I can also deploy it on Monday morning.

It is fine on Monday morning. It is 7pm here, it has been a busy week so I am going to take care of the family :-] But I will be there on Monday at 9:00 CEST as usual!

In T274940#6949300, @Raymond wrote:

In T274940#6949275, @hashar wrote:

I have added T278562 as a subtask since that is related to the roll out of 1.36.0-wmf.36. mediawiki/core has a feature relying on the wiki page MediaWiki:Autoblock_whitelist and it got proposed for renaming. That has been done on commonswiki after .36 got deployed but when we rolled back to .35 the code obviously did not know about the new page name. It is hard :]

T278562 is restricted for me. Maybe the following is not related: I restored the content of the old message MediaWiki:Autoblock_whitelist this morning after I read about an issue: https://commons.wikimedia.org/wiki/Commons:Village_pump#CropTool_returns_%22Blocked_user%22_error_message

I added you into the list of people who can view that task. It will be made public next week.

I have pushed 1.36.0-wmf.36 to group 1 wikis this morning at 8:00 UTC. Given we had it for a full day last week and the blockers got addressed.

I wanted to push it to the rest of the wikis notably for the FlaggedRevs (T278478) issue. That one has been confirmed to be fixed via mwdebug by @Ladsgroup. However @Urbanecm remembered me that today March 29th is a no-deploy day since it is a WMF granted holiday.

So postponed, I will push .36 to the rest of the wikis on Tuesday morning around 7:00 UTC.

Mentioned in SAL (#wikimedia-operations) [2021-03-30T07:28:44Z] <hashar@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.36 - T274940