Page MenuHomePhabricator

1.36.0-wmf.34 deployment blockers
Closed, ResolvedPublicRelease

Details

Backup Conductor
LarsWirzenius
Release Version
1.36.0-wmf.34
Release Date
Mar 8 2021, 12:00 AM

2021 week 10 1.36-wmf.34 Changes wmf/1.36.0-wmf.34

This MediaWiki Train Deployment is scheduled for the week of Monday, March 8th:

Monday March 8thTuesday, March 9thWednesday, March 10thThursday, March 11thFriday
Backports only.Branch wmf.34 and deploy to Group 0 Wikis.Deploy wmf.34 to Group 1 Wikis.Deploy wmf.34 to all Wikis.No deployments on fridays

How this works

  • Any serious bugs affecting wmf.34 should be added as subtasks beneath this one.
  • Any open subtask(s) block the train from moving forward.This means no further deployments until the blockers are resolved.
  • If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
  • For more info about deployment blockers, see Holding the train.

Related Links

Other Deployments

Previous: 1.36.0-wmf.33
Next: 1.36.0-wmf.35

Related Objects

Event Timeline

Change 670019 had a related patch set uploaded (by DannyS712; owner: trainbranchbot):
[mediawiki/core@wmf/1.36.0-wmf.34] Branch commit for wmf/1.36.0-wmf.34

https://gerrit.wikimedia.org/r/670019

Change 670019 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.34] Branch commit for wmf/1.36.0-wmf.34

https://gerrit.wikimedia.org/r/670019

Actions to date:

  • Patched a copy of logspam-watch to consolidate "Prefix search request was longer than the maximum allowed length" errors.
  • As an experiment, running today's actions under script ~/1.36.0-wmf.34.log
    • This will be messy output, but might be nice to have if there's an incident to document.
  • 16:31 UTC: +2 on branch commit, merged at 17:03.
  • 17:07 - 17:16: Backported Session tick: Add data QA flag if the user is in group data-qa to wmf/1.36.0-wmf.34
  • 17:22: scap prep
  • 17:35 - 18:00:
    • scap apply-patches --train 1.36.0-wmf.34 fails on patches requiring --3way
    • Updated relevant security tickets, confirmed with security that patches should still be good
    • Proceeding under the assumption, from discussion on IRC and today's comments on T269153, that --3way is probably fine in practice but should be raised on relevant tickets when it comes up. (That understanding might need refined, it's just what I landed on with a serious case of train brain happening.)
  • 17:57: deploy-promote to testwikis, pressing enter to continue
  • 18:01: Mentioned train on #tech-department in Element / Slack

Ran scap clean --delete 1.36.0-wmf.31. Some errors there:

cannot delete non-empty directory: php-1.36.0-wmf.31/cache/l10n
cannot delete non-empty directory: php-1.36.0-wmf.31/cache/l10n
cannot delete non-empty directory: php-1.36.0-wmf.31/cache
cannot delete non-empty directory: php-1.36.0-wmf.31/cache
cannot delete non-empty directory: php-1.36.0-wmf.31
cannot delete non-empty directory: php-1.36.0-wmf.18/cache/l10n
cannot delete non-empty directory: php-1.36.0-wmf.18/cache/l10n
cannot delete non-empty directory: php-1.36.0-wmf.18/cache
cannot delete non-empty directory: php-1.36.0-wmf.18/cache
cannot delete non-empty directory: php-1.36.0-wmf.18

This rings a bell but I don't remember what causes it. Noting here to sort out later.

Mentioned in SAL (#wikimedia-operations) [2021-03-09T20:09:22Z] <brennen> train status: 1.36.0-wmf.32 (T274938) on group0 at 20:06:32 UTC; logs initially quiet.

End-of-day status: All quiet since deploy to group0.

Mentioned in SAL (#wikimedia-operations) [2021-03-10T19:58:23Z] <brennen> train status: 1.36.0-wmf.34 (T274938): currently blocked at group0 as client error logging is broken (UBN ticket incoming), will hold for patch.

First patch for T277094 didn't work out. @Jdlrobson continues to investigate; holding train until we have a resolution for the client error situation.

Sent blocker mail just now, for completeness. Also a brief status update to #tech-department on WMF Slack.

Mentioned in SAL (#wikimedia-operations) [2021-03-10T21:30:08Z] <brennen> train status: 1.36.0-wmf.34 (T274938): logstash client error board was set up incorrectly; reverting earlier patch for T277094 and will proceed to group1.

Mentioned in SAL (#wikimedia-operations) [2021-03-10T21:43:22Z] <brennen> train status: 1.36.0-wmf.34 (T274938): client errors may still be missing for group0; continuing to hold for T277094 until we know what's broken.

Mentioned in SAL (#wikimedia-operations) [2021-03-10T22:26:37Z] <brennen> train status: 1.36.0-wmf.34 (T274938): T277094 believed resolved, promoting to group1.

End-of-day notes:

  • Log triage meeting was largely uneventful, we reported a few things from previous deploy cycles and cleared a gratifying number of logstash filters for things that had been fixed.
  • It's really nice to have Phatality working again.
  • wmf.34 has been stable on group1 since deploy.

Mentioned in SAL (#wikimedia-operations) [2021-03-11T21:20:25Z] <brennen> train status: 1.36.0-wmf.34 (T274938): rolling back to group1 and marking T277229 a train blocker

End of day status:

  • On all wikis since 22:41 UTC
  • Things seem pretty stable as far as PHP error logs go.
  • I think there's an active wikimedia-client-errors-alerts alert, but things in mw-client-errors look quiet unless the NOT file_url: undefined filter is disabled. Plausibly this is more or less an artifact?

I missed a full stop. Will deal with this Monday. We will have to live with the active alert until then. Sorry about that.

No worries - and thanks for the assistance this week. I'm signing off for the day. As usual, will mark this resolved if we get through Friday without a rollback.

Note: Removing T277302 as wmf.34 is fully deployed at this point and it's been added as blocker for wmf.35.