Page MenuHomePhabricator

1.39.0-wmf.12 deployment blockers
Closed, ResolvedPublic5 Estimated Story PointsRelease

Details

Backup Train Conductor
hashar
Release Version
1.39.0-wmf.12
Release Date
May 16 2022, 12:00 AM

2022 week 20 1.39-wmf.12 Changes wmf/1.39.0-wmf.12

This MediaWiki Train Deployment is scheduled for the week of Monday, May 16th:

Monday May 16thTuesday, May 17thWednesday, May 18thThursday, May 19thFriday
Backports only.Branch wmf.12 and deploy to Group 0 Wikis.Deploy wmf.12 to Group 1 Wikis.Deploy wmf.12 to all Wikis.No deployments on fridays

How this works

  • Any serious bugs affecting wmf.12 should be added as subtasks beneath this one.
  • Any open subtask(s) block the train from moving forward. This means no further deployments until the blockers are resolved.
  • If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
  • If you have a risky change in this week's train add a comment to this task using the Risky patch template
  • For more info about deployment blockers, see Holding the train.

Related Links

Other Deployments

Previous: 1.39.0-wmf.11
Next: 1.39.0-wmf.13

Event Timeline

Risky Patch! 🚂🔥
thcipriani triaged this task as Medium priority.
thcipriani updated Other Assignee, added: hashar.
thcipriani set the point value for this task to 5.
Risky Patch! 🚂🔥
  • Change: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/767769
  • Summary: Changing connection management to the databases
  • Test plan: Errors related to database should not go up.
  • Places to monitor:
  • Revert plan: Probably rolling back train would be ok. Quick fix in cases would be to replace getConnection with getConnectionInternal and the patch better not to be reverted.
  • Affected wikis: all
  • IRC contact: Amir1, Krinkle
  • UBN Task Projects/tags: DBA Performance-Team
  • Would you like to backport this change rather than ride the train?: No because I want it to be tested in group0 first.

Change 792557 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792557

Change 792557 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792557

Mentioned in SAL (#wikimedia-operations) [2022-05-17T07:22:07Z] <jnuche@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.12 refs T305218

Mentioned in SAL (#wikimedia-operations) [2022-05-17T07:39:23Z] <jnuche@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.12 refs T305218

Mentioned in SAL (#wikimedia-operations) [2022-05-17T07:53:58Z] <jnuche@deploy1002> Finished scap: testwikis wikis to 1.39.0-wmf.12 refs T305218 (duration: 14m 35s)

Change 792563 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792563

Change 792563 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792563

Mentioned in SAL (#wikimedia-operations) [2022-05-17T08:06:02Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.12 refs T305218

Change 792976 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792976

Change 792976 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792976

Change 792978 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792978

Change 792978 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/792978

Mentioned in SAL (#wikimedia-operations) [2022-05-18T08:26:06Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.12 refs T305218

Mentioned in SAL (#wikimedia-operations) [2022-05-18T08:27:01Z] <jnuche@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.12 refs T305218 (duration: 00m 53s)

There is a huge spike of "MediaWiki\\Parser\\ParserObserver::notifyParse: Possibly redundant parse!","type":"mediawiki","channel":"DuplicateParse" errors in production. These are not visible on the new-errors dashboard due to the large message payload being truncated, and thus not indexed by OpenSearch, but they are instead visible on the mediawiki-errors dashboard.

The start of this correlates exactly with the 09:00 UTC rollout of group1 wikis to 1.39.0-wmf.12.

I believe this is train-blocking for reasons of exessive logging and making telemetry unstable/unusable. Secondarily, the warning itselfg may also be significant and train-blocking, depending on what investigation finds. Tertiarily, the warning should be fixed post-hoc to not exceed the allowed payload size, but that is not urgent.

Screenshot (739×1 px, 111 KB)
Screenshot (663×1 px, 97 KB)

This one was probably caused by yours truly: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/791702

I think I know what part caused it, we probably should replace it with hash

Change 793120 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] ParserObserver: Avoid pushing the whole content to the debug log

https://gerrit.wikimedia.org/r/793120

Change 792665 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.39.0-wmf.12] parser: Avoid pushing the whole content to ParserObserver debug log

https://gerrit.wikimedia.org/r/792665

Change 792665 merged by jenkins-bot:

[mediawiki/core@wmf/1.39.0-wmf.12] parser: Avoid pushing the whole content to ParserObserver debug log

https://gerrit.wikimedia.org/r/792665

Mentioned in SAL (#wikimedia-operations) [2022-05-18T22:30:03Z] <ladsgroup@deploy1002> Synchronized php-1.39.0-wmf.12/includes/parser/ParserObserver.php: Backport: [[gerrit:792665|parser: Avoid pushing the whole content to ParserObserver debug log (T305218)]] (duration: 00m 52s)

Change 793120 merged by jenkins-bot:

[mediawiki/core@master] parser: Avoid pushing the whole content to ParserObserver debug log

https://gerrit.wikimedia.org/r/793120

The json truncated thing is fixed now:

image.png (380×979 px, 42 KB)

Change 793476 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/793476

Change 793476 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.12 refs T305218

https://gerrit.wikimedia.org/r/793476

Mentioned in SAL (#wikimedia-operations) [2022-05-19T13:52:49Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.12 refs T305218