Page MenuHomePhabricator

🧪🚂 Trainsperiment Week: 1.39.0-wmf.1, 1.39.0-wmf.2, 1.39.0-wmf.3, 1.39.0-wmf.4 deployment blockers
Closed, ResolvedPublicRelease

Description

Blocker for 1.39.0-wmf.1–1.39.0-wmf.4

See 🧪🚂 Trainsperiment Week MediaWiki page for schedule and details

Details

Release Date
Mar 21 2022, 12:00 AM

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2022-03-21T16:58:24Z] <brennen> trainsperiment (T300203): blockers currently cleared, will hold wmf.1 -> group2 until 18:00 UTC, per deployment calendar

Change 772450 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.1 refs T300203

https://gerrit.wikimedia.org/r/772450

Change 772450 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.1 refs T300203

https://gerrit.wikimedia.org/r/772450

Mentioned in SAL (#wikimedia-operations) [2022-03-21T18:04:37Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.1 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-21T18:19:36Z] <brennen> trainsperiment (T300203): 1.39.0-wmf.1 on all wikis; starting prep of wmf.2, will abort if needed

Change 772454 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772454

Change 772454 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772454

Mentioned in SAL (#wikimedia-operations) [2022-03-21T18:22:23Z] <brennen@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.2 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-21T19:26:57Z] <brennen@deploy1002> Finished scap: testwikis wikis to 1.39.0-wmf.2 refs T300203 (duration: 64m 33s)

Change 772494 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772494

Change 772494 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772494

Mentioned in SAL (#wikimedia-operations) [2022-03-21T19:42:29Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.2 refs T300203

Change 772501 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772501

Change 772501 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772501

Mentioned in SAL (#wikimedia-operations) [2022-03-21T20:49:04Z] <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.2 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-21T20:49:55Z] <dduvall@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.2 refs T300203 (duration: 00m 51s)

Change 772503 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772503

Change 772503 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772503

Mentioned in SAL (#wikimedia-operations) [2022-03-21T20:56:18Z] <dduvall@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.2 refs T300203

Change 772789 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772789

Change 772789 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772789

Mentioned in SAL (#wikimedia-operations) [2022-03-22T08:52:26Z] <jnuche@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.3 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-22T09:54:34Z] <jnuche@deploy1002> Finished scap: testwikis wikis to 1.39.0-wmf.3 refs T300203 (duration: 62m 07s)

Change 772794 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772794

Change 772794 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772794

Mentioned in SAL (#wikimedia-operations) [2022-03-22T10:03:37Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.3 refs T300203

The train is blocked by scap not flushing the PHP opcache on all the application servers. It only process 86 of them, should be a few hundred. T304414

Change 772826 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772826

Change 772826 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772826

Mentioned in SAL (#wikimedia-operations) [2022-03-22T13:26:07Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.3 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-22T13:27:00Z] <jnuche@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.3 refs T300203 (duration: 00m 52s)

Change 772830 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772830

Change 772830 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772830

Mentioned in SAL (#wikimedia-operations) [2022-03-22T13:39:55Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.3 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-22T17:25:07Z] <brennen> trainsperiment (T300203): with 1.39.0-wmf.3 on all wikis, we're paused for a planned catchup window - nothing to do at the moment, we'll deploy 1.39.0-wmf.4 tomorrow (2022-03-23).

Adding T304476 as a blocker for wmf4 (currently in wmf3).

Mentioned in SAL (#wikimedia-operations) [2022-03-23T17:48:15Z] <brennen> trainsperiment (T300203): starting prep for 1.39.0-wmf.4

Change 773293 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773293

Change 773293 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773293

Mentioned in SAL (#wikimedia-operations) [2022-03-23T17:52:32Z] <brennen@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.4 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-23T18:42:13Z] <brennen@deploy1002> Finished scap: testwikis wikis to 1.39.0-wmf.4 refs T300203 (duration: 49m 41s)

Mentioned in SAL (#wikimedia-operations) [2022-03-23T18:47:35Z] <brennen> trainsperiment (T300203): 1.39.0-wmf.4 on testwikis; proceeding to groups 0-2 with 15 minute intervals for watching logs

Mentioned in SAL (#wikimedia-operations) [2022-03-23T18:53:07Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.4 refs T300203

Change 773304 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773304

Change 773304 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773304

Mentioned in SAL (#wikimedia-operations) [2022-03-23T18:59:21Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.4 refs T300203

Change 773326 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773326

Change 773326 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773326

Mentioned in SAL (#wikimedia-operations) [2022-03-23T19:08:14Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.4 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-23T19:09:07Z] <brennen@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.4 refs T300203 (duration: 00m 52s)

Change 773330 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773330

Change 773330 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773330

Mentioned in SAL (#wikimedia-operations) [2022-03-23T19:23:36Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.4 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-23T19:37:24Z] <brennen> trainsperiment (T300203): 1.39.0-wmf.4 on all wikis; logs seem clean - end of train deployment activities for the week, unless bugs emerge

Mentioned in SAL (#wikimedia-operations) [2022-03-23T23:34:02Z] <brennen> trainsperiment (T300203): reverting to 1.39.0-wmf.3 on all wikis for T304564; will move forward again after a fix.

Change 773363 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/773363

Change 773363 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/773363

Mentioned in SAL (#wikimedia-operations) [2022-03-23T23:38:02Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.3 refs T300203

End-of-workday notes: See T304564: MWException: `[title]` is not a valid file title.; once that's patched the train can move forward again.

I think we're in a fairly good place. wmf.4 is only about a day's worth of delta from wmf.3, and has been on all wikis for most of the US afternoon.

T304564 fix has been backported and deployed

T304559 concerns RevisionStore I have poke platform engineering team about

T304559 concerns RevisionStore I have poke platform engineering team about

Daniel and Lucas spoke about earlier today and given the rare occurrence we have decided it is not worth keeping it as a train blocker. T304559#7802768

Mentioned in SAL (#wikimedia-operations) [2022-03-24T16:13:27Z] <brennen> trainsperiment (T300203): blockers clear, logs triaged, rolling 1.39.0-wmf.4 out to all wikis again

Change 773579 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773579

Change 773579 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773579

Mentioned in SAL (#wikimedia-operations) [2022-03-24T16:20:33Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.4 refs T300203

Change 773582 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773582

Change 773582 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773582

Mentioned in SAL (#wikimedia-operations) [2022-03-24T16:24:17Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.4 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-24T16:25:25Z] <brennen@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.4 refs T300203 (duration: 01m 06s)

Change 773583 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773583

Change 773583 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773583

Mentioned in SAL (#wikimedia-operations) [2022-03-24T16:29:53Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.4 refs T300203

thcipriani claimed this task.