Page MenuHomePhabricator

1.40.0-wmf.13 deployment blockers
Closed, ResolvedPublic5 Estimated Story PointsRelease

Details

Backup Train Conductor
dancy
Release Version
1.40.0-wmf.13
Release Date
Dec 5 2022, 12:00 AM

2022 week 49 1.40-wmf.13 Changes wmf/1.40.0-wmf.13

This MediaWiki Train Deployment is scheduled for the week of Monday, December 5th:

Monday December 5thTuesday, December 6thWednesday, December 7thThursday, December 8thFriday
Backports only.Branch wmf.13 and deploy to Group 0 Wikis.Deploy wmf.13 to Group 1 Wikis.Deploy wmf.13 to all Wikis.No deployments on fridays

How this works

  • Any serious bugs affecting wmf.13 should be added as subtasks beneath this one.
  • Any open subtask(s) block the train from moving forward. This means no further deployments until the blockers are resolved.
  • If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
  • If you have a risky change in this week's train add a comment to this task using the Risky patch template
  • For more info about deployment blockers, see Holding the train.

Related Links

Other Deployments

Previous: 1.40.0-wmf.12
Next: 1.40.0-wmf.14

Event Timeline

thcipriani triaged this task as Medium priority.
thcipriani updated Other Assignee, added: dancy.
thcipriani set the point value for this task to 5.

Change 864819 had a related patch set uploaded (by TrainBranchBot; author: trainbranchbot):

[mediawiki/core@wmf/1.40.0-wmf.13] Branch commit for wmf/1.40.0-wmf.13

https://gerrit.wikimedia.org/r/864819

Change 864819 merged by jenkins-bot:

[mediawiki/core@wmf/1.40.0-wmf.13] Branch commit for wmf/1.40.0-wmf.13

https://gerrit.wikimedia.org/r/864819

Change 864888 had a related patch set uploaded (by TrainBranchBot; author: MediaWiki PreSync):

[operations/mediawiki-config@master] testwikis wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/864888

Change 864888 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/864888

Mentioned in SAL (#wikimedia-operations) [2022-12-06T04:02:28Z] <mwpresync@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.13 refs T320518

Mentioned in SAL (#wikimedia-operations) [2022-12-06T12:14:54Z] <jnuche@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.13 refs T320518

Mentioned in SAL (#wikimedia-operations) [2022-12-06T12:21:26Z] <jnuche@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.13 refs T320518

Mentioned in SAL (#wikimedia-operations) [2022-12-06T12:27:18Z] <jnuche@deploy1002> Finished scap: testwikis wikis to 1.40.0-wmf.13 refs T320518 (duration: 05m 52s)

Not sure if T323542: Forbid new modern code from not targeting mobile should have been a train blocker, but if it was, it no longer is: the revert is merged into wmf.13.

FYI https://gerrit.wikimedia.org/r/865128 will need to be merged before the train can continue.

Change 865154 had a related patch set uploaded (by TrainBranchBot; author: Chad Horohoe):

[operations/mediawiki-config@master] group0 wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/865154

Change 865154 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/865154

Mentioned in SAL (#wikimedia-operations) [2022-12-06T19:19:35Z] <demon@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.40.0-wmf.13 refs T320518

Mentioned in SAL (#wikimedia-operations) [2022-12-07T13:34:45Z] <mwpresync@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.13 refs T320518

Mentioned in SAL (#wikimedia-operations) [2022-12-07T13:42:30Z] <mwpresync@deploy1002> Finished scap: testwikis wikis to 1.40.0-wmf.13 refs T320518 (duration: 07m 45s)

Change 865733 had a related patch set uploaded (by TrainBranchBot; author: Chad Horohoe):

[operations/mediawiki-config@master] group1 wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/865733

Change 865733 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/865733

Mentioned in SAL (#wikimedia-operations) [2022-12-07T20:13:49Z] <demon@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.40.0-wmf.13 refs T320518

Mentioned in SAL (#wikimedia-operations) [2022-12-07T20:20:53Z] <demon@deploy1002> Synchronized php: group1 wikis to 1.40.0-wmf.13 refs T320518 (duration: 07m 03s)

Change 866498 had a related patch set uploaded (by TrainBranchBot; author: Chad Horohoe):

[operations/mediawiki-config@master] group2 wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/866498

Change 866498 merged by jenkins-bot:

[operations/mediawiki-config@master] group2 wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/866498

Mentioned in SAL (#wikimedia-operations) [2022-12-08T20:12:26Z] <demon@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.40.0-wmf.13 refs T320518

We're seeing a 3x increase in throttled logs since this rolled out. I suspect it's from Module "{module}" not loadable on target "{target}". (logstash) but can't be sure because the throttler is fully engaged on these messages since 2022-12-07@20:00Z.

To protect the Logstash cluster, we start dropping logs when the cluster sees 100 duplicate logs per second matching the template <type><channel><level><normalized_message>.

@Jdlrobson I think it's due to the module target changes.

Is it because a module is marked to target mobile but something in the tree is not? If that's the case, I think I can write a script to go through all of RL modules dependency tree and find the ones that can be easily converted without causing issues.

Didn't "mediawiki.action.view.metadata" switch? https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2022.12.08?id=SnDA84QBIFwrQt9tiW67 the only dependency is mediawiki.action.view.filepage which got fixed in the same patch

Ah, it seems the patches didn't make it to branch cut. Backporting.

Change 866469 had a related patch set uploaded (by Ladsgroup; author: Jdlrobson):

[mediawiki/core@wmf/1.40.0-wmf.13] File pages: Add mobile targets to modules that are silently being removed

https://gerrit.wikimedia.org/r/866469

Change 866469 merged by jenkins-bot:

[mediawiki/core@wmf/1.40.0-wmf.13] File pages: Add mobile targets to modules that are silently being removed

https://gerrit.wikimedia.org/r/866469

Mentioned in SAL (#wikimedia-operations) [2022-12-08T23:02:10Z] <ladsgroup@deploy1002> Started scap: Backport for [[gerrit:866469|File pages: Add mobile targets to modules that are silently being removed (T324723 T320518)]]

Mentioned in SAL (#wikimedia-operations) [2022-12-08T23:03:58Z] <ladsgroup@deploy1002> ladsgroup and ladsgroup: Backport for [[gerrit:866469|File pages: Add mobile targets to modules that are silently being removed (T324723 T320518)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-12-08T23:11:23Z] <ladsgroup@deploy1002> Finished scap: Backport for [[gerrit:866469|File pages: Add mobile targets to modules that are silently being removed (T324723 T320518)]] (duration: 09m 12s)

It's in a much better state now:
https://grafana.wikimedia.org/d/000000561/logstash?from=1670529261667&orgId=1&to=1670542372306&viewPanel=45

image.png (968×1 px, 174 KB)

It'll be cleaned up more in the future. If it's still causing issues, ping us to start sampling.

The drop rate has settled to a 2x increase since the deployment. Thanks @Ladsgroup!

Are these logs revealing a deeper issue that is being worked on? The logging pipeline is not in immediate danger, but we would prefer to avoid normalizing the deviance.

Hey @colewhite I need a week's worth of data to address a deeper issue. I'm going to turn this logging off next week in time for the holidays. Feel free to +2 https://gerrit.wikimedia.org/r/c/mediawiki/core/+/864722 as I wasn't sure if I can self merge that :-)

The train was rolled back because of T324801#8455874.

The blocker had a patch which needs to be backported. I can run the train at 13:00 UTC to move forward.

Change 866586 had a related patch set uploaded (by TrainBranchBot; author: Hashar):

[operations/mediawiki-config@master] all wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/866586

Change 866586 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.40.0-wmf.13

https://gerrit.wikimedia.org/r/866586

Mentioned in SAL (#wikimedia-operations) [2022-12-09T13:13:16Z] <hashar@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.40.0-wmf.13 refs T320518

Hey @colewhite I need a week's worth of data to address a deeper issue. I'm going to turn this logging off next week in time for the holidays.

Sounds like a plan. Thanks!

Mentioned in SAL (#wikimedia-operations) [2022-12-09T17:03:31Z] <claime> eventgate-analytics bumped to 30 replicas to absorb increased load - T320518