Change 670019 had a related patch set uploaded (by DannyS712; owner: trainbranchbot):
[mediawiki/core@wmf/1.36.0-wmf.34] Branch commit for wmf/1.36.0-wmf.34

https://gerrit.wikimedia.org/r/670019

Mentioned in SAL (#wikimedia-operations) [2021-03-09T16:31:04Z] <brennen> 1.36.0-wmf.34 was branched at e175899921535f83e168145cbe942489475607db for T274938

Change 670019 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.34] Branch commit for wmf/1.36.0-wmf.34

https://gerrit.wikimedia.org/r/670019

Actions to date:

• Patched a copy of logspam-watch to consolidate "Prefix search request was longer than the maximum allowed length" errors.
• As an experiment, running today's actions under script ~/1.36.0-wmf.34.log
  • This will be messy output, but might be nice to have if there's an incident to document.
• 16:31 UTC: +2 on branch commit, merged at 17:03.
• 17:07 - 17:16: Backported Session tick: Add data QA flag if the user is in group data-qa to wmf/1.36.0-wmf.34
  • backport
• 17:22: scap prep
• 17:35 - 18:00:
  • scap apply-patches --train 1.36.0-wmf.34 fails on patches requiring --3way
  • Updated relevant security tickets, confirmed with security that patches should still be good
  • Proceeding under the assumption, from discussion on IRC and today's comments on T269153, that --3way is probably fine in practice but should be raised on relevant tickets when it comes up. (That understanding might need refined, it's just what I landed on with a serious case of train brain happening.)
• 17:57: deploy-promote to testwikis, pressing enter to continue
• 18:01: Mentioned train on #tech-department in Element / Slack

Ran scap clean --delete 1.36.0-wmf.31. Some errors there:

cannot delete non-empty directory: php-1.36.0-wmf.31/cache/l10n
cannot delete non-empty directory: php-1.36.0-wmf.31/cache/l10n
cannot delete non-empty directory: php-1.36.0-wmf.31/cache
cannot delete non-empty directory: php-1.36.0-wmf.31/cache
cannot delete non-empty directory: php-1.36.0-wmf.31
cannot delete non-empty directory: php-1.36.0-wmf.18/cache/l10n
cannot delete non-empty directory: php-1.36.0-wmf.18/cache/l10n
cannot delete non-empty directory: php-1.36.0-wmf.18/cache
cannot delete non-empty directory: php-1.36.0-wmf.18/cache
cannot delete non-empty directory: php-1.36.0-wmf.18

This rings a bell but I don't remember what causes it. Noting here to sort out later.

Mentioned in SAL (#wikimedia-operations) [2021-03-09T20:09:22Z] <brennen> train status: 1.36.0-wmf.32 (T274938) on group0 at 20:06:32 UTC; logs initially quiet.

End-of-day status: All quiet since deploy to group0.

Mentioned in SAL (#wikimedia-operations) [2021-03-10T19:58:23Z] <brennen> train status: 1.36.0-wmf.34 (T274938): currently blocked at group0 as client error logging is broken (UBN ticket incoming), will hold for patch.

First patch for T277094 didn't work out. @Jdlrobson continues to investigate; holding train until we have a resolution for the client error situation.

Sent blocker mail just now, for completeness. Also a brief status update to #tech-department on WMF Slack.

Mentioned in SAL (#wikimedia-operations) [2021-03-10T21:30:08Z] <brennen> train status: 1.36.0-wmf.34 (T274938): logstash client error board was set up incorrectly; reverting earlier patch for T277094 and will proceed to group1.

Mentioned in SAL (#wikimedia-operations) [2021-03-10T21:43:22Z] <brennen> train status: 1.36.0-wmf.34 (T274938): client errors may still be missing for group0; continuing to hold for T277094 until we know what's broken.

Mentioned in SAL (#wikimedia-operations) [2021-03-10T22:26:37Z] <brennen> train status: 1.36.0-wmf.34 (T274938): T277094 believed resolved, promoting to group1.

End-of-day notes:

• Log triage meeting was largely uneventful, we reported a few things from previous deploy cycles and cleared a gratifying number of logstash filters for things that had been fixed.
• It's really nice to have Phatality working again.
• wmf.34 has been stable on group1 since deploy.

Mentioned in SAL (#wikimedia-operations) [2021-03-11T21:20:25Z] <brennen> train status: 1.36.0-wmf.34 (T274938): rolling back to group1 and marking T277229 a train blocker

Mentioned in SAL (#wikimedia-operations) [2021-03-11T22:36:41Z] <brennen> train status: 1.36.0-wmf.34 (T274938): T277229 and T266517 related issues hopefully resolved, rolling forward to all wikis

End of day status:

• On all wikis since 22:41 UTC
• Things seem pretty stable as far as PHP error logs go.
• I think there's an active wikimedia-client-errors-alerts alert, but things in mw-client-errors look quiet unless the NOT file_url: undefined filter is disabled. Plausibly this is more or less an artifact?

I missed a full stop. Will deal with this Monday. We will have to live with the active alert until then. Sorry about that.

No worries - and thanks for the assistance this week. I'm signing off for the day. As usual, will mark this resolved if we get through Friday without a rollback.