1.37.0-wmf.14 deployment blockers
Closed, ResolvedPublicRelease
Actions

Assigned To

Authored By

	thcipriani
	Apr 26 2021, 3:15 PM

Details

Backup Train Conductor: brennen
Release Version: 1.37.0-wmf.14
Release Date: Jul 12 2021, 12:00 AM

2021 week 28 1.37-wmf.14 Changes wmf/1.37.0-wmf.14

This MediaWiki Train Deployment is scheduled for the week of Monday, July 12th:

Monday July 12th	Tuesday, July 13th	Wednesday, July 14th	Thursday, July 15th	Friday
Backports only.	Branch `wmf.14` and deploy to Group 0 Wikis.	Deploy `wmf.14` to Group 1 Wikis.	Deploy `wmf.14` to all Wikis.	No deployments on fridays

See https://wikitech.wikimedia.org/wiki/Deployments for full schedule.

How this works

Any serious bugs affecting wmf.14 should be added as subtasks beneath this one.
- Use this form to create one.
Any open subtask(s) block the train from moving forward. This means no further deployments until the blockers are resolved.
If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
If you have a risky change in this week's train add a comment to this task using the Risky patch template
For more info about deployment blockers, see Holding the train.

Other Deployments

Previous: 1.37.0-wmf.13

Next: 1.37.0-wmf.15

Related Objects

Mentioned In: T286679: Some interface messages are in Welsh when British English is selected as the interface language
Mentioned Here: T286679: Some interface messages are in Welsh when British English is selected as the interface language
T286521: Deadlock found when trying to get lock (UserOptionsManager::saveOptionsQuery)
T286463: mw2383 is misbehaving
T285820: Switch deployment server to codfw (July 2021)

Event Timeline

thcipriani created this task.Apr 26 2021, 3:15 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 26 2021, 3:15 PM

thcipriani assigned this task to brennen.Jun 6 2021, 8:36 PM

thcipriani triaged this task as Medium priority.

thcipriani updated Other Assignee, added: • LarsWirzenius.

thcipriani added a project: Release-Engineering-Team (Doing).

thcipriani reassigned this task from brennen to dancy.Jun 10 2021, 4:32 PM

thcipriani updated Other Assignee, added: brennen; removed: • LarsWirzenius.

thcipriani added a subscriber: brennen.

Note T285820: Deployment server is likely to be switched to deploy2002 prior to train on 2021-07-14.

Krinkle added a subtask: T286040: "InvalidArgumentException: $key must be a string or an array" on beta login and vote wikis.Jul 2 2021, 4:28 PM

Ladsgroup subscribed.Jul 3 2021, 4:14 PM

DannyS712 added a subtask: T286140: AbuseLog no longer recording revids of saved edits.Jul 4 2021, 5:15 AM

brennen removed a subtask: T286140: AbuseLog no longer recording revids of saved edits.Jul 4 2021, 5:49 PM

Risky Patch! 🚂🔥

Change: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/697641
Summary:
- This patch is trivial but it touches a lot of code, thus - risky. I moves the CSRF token generation out of User object into a new CsrfTokenSet class. If something goes wrong users CSRF tokens won't match and edits/uploads/basically any write operation will be rejected.
Test plan:
- Did a lot of manual testing, but again, it touches almost every feature, so something might have been missed.
Places to monitor:
- Logstash: mediawiki-errors
- Grafana: mediawiki-errors
Revert plan: Revert the patch
Affected wikis:
- All wikis
IRC contact: Pchelolo
UBN Task Projects/tags: Platform Team Workboards (MW Expedition)

Revert plan: Revert the patch

I think the most likely issue is a typo or copy/paste error (inverted logic, wrong token name, missing parameter). The fix should be fairly obvious from the diff, so perhaps the first thing to try would be to simply fix the issue manually rather than revert the patch.

Note: During sync, you're likely to see something like:

17:04:22 ['/usr/bin/scap', 'pull', '--no-update-l10n', '--no-php-restart', '--include', 'php-1.37.0-wmf.12', '--include', 'php-1.37.0-wmf.12/includes', '--include', 'php-1.37.0-wmf.12/includes/user', '--include', 'php-1.37.0-wmf.12/includes/user/UserOptionsManager.php', 'mw1319.eqiad.wmnet', 'mw1366.eqiad.wmnet', 'mw2300.codfw.wmnet', 'mw2289.codfw.wmnet', 'mw2254.codfw.wmnet', 'mw1269.eqiad.wmnet', 'mw1285.eqiad.wmnet', 'mw1313.eqiad.wmnet'] on mw2383.codfw.wmnet returned [255]: ssh: connect to host mw2383.codfw.wmnet port 22: Connection timed out
sync-apaches: 100% (ok: 342; fail: 1; left: 0)
17:04:22 1 apaches had sync errors

...

17:05:01 /usr/bin/sudo -u root -- /usr/local/sbin/check-and-restart-php php7.2-fpm 100 on mw2383.codfw.wmnet returned [255]: ssh: connect to host mw2383.codfw.wmnet port 22: Connection timed out

17:05:01 1 hosts had failures restarting php-fpm

As mw2383 is misbehaving - see T286463.

@brennen Thanks for the heads-up.

Jdlrobson removed a subtask: T286040: "InvalidArgumentException: $key must be a string or an array" on beta login and vote wikis.Jul 13 2021, 8:06 PM

.14 has been deployed to testwikis and is now blocked pending resolution of T286521.

Late (US) Wednesday afternoon status: After work on T286521, we've rolled forward to group1. Generally, things seem quiet, although T286521 continues to crop up.

Possible blocker: T286679: Some interface messages are in Welsh when British English is selected as the interface language isn’t a very serious issue, but may annoy a number of English Wikipedia users if it reaches group2. (It’s not yet clear how many messages would be affected, and I also don’t know how many people actually have their user interface set to en-gb.)

brennen removed a subtask: T286521: Deadlock found when trying to get lock (UserOptionsManager::saveOptionsQuery).Jul 14 2021, 10:35 PM