🧪🚂 Trainsperiment Week: 1.39.0-wmf.1, 1.39.0-wmf.2, 1.39.0-wmf.3, 1.39.0-wmf.4 deployment blockers
Closed, ResolvedPublicRelease
Actions

Description

Blocker for 1.39.0-wmf.1–1.39.0-wmf.4

See 🧪🚂 Trainsperiment Week MediaWiki page for schedule and details

Details

Release Date: Mar 21 2022, 12:00 AM

	Project	Branch	Lines +/-	Subject
	mediawiki/core	wmf/1.39.0-wmf.1	+981 -3	Branch commit for wmf/1.39.0-wmf.1

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved	Release	thcipriani	T300203 🧪🚂 Trainsperiment Week: 1.39.0-wmf.1, 1.39.0-wmf.2, 1.39.0-wmf.3, 1.39.0-wmf.4 deployment blockers
Resolved	PRODUCTION ERROR	aaron	T281451 Wikimedia\Rdbms\DBTransactionError: Transaction round stage must be 'cursory' (not 'within-rollback-callbacks')
Resolved		aaron	T297424 Reduce LBFactory::rollbackPrimaryChanges() callers in core
Resolved	PRODUCTION ERROR	hashar	T304307 Wikimedia\Timestamp\TimestampException: Wikimedia\Timestamp\ConvertibleTimestamp::getTimestamp: The timestamp cannot be represented in the specified format
Resolved	PRODUCTION ERROR	taavi	T304331 PageSplitterHooks: Error: Call to a member function exists() on null
Resolved		Joe	T304414 scap no more restart php opcache on all servers
Resolved	PRODUCTION ERROR	Arlolra	T304564 MWException: `[title]` is not a valid file title.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2022-03-21T16:58:24Z] <brennen> trainsperiment (T300203): blockers currently cleared, will hold wmf.1 -> group2 until 18:00 UTC, per deployment calendar

Zabe closed subtask T304331: PageSplitterHooks: Error: Call to a member function exists() on null as Resolved.Mar 21 2022, 5:00 PM

Change 772450 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.1 refs T300203

https://gerrit.wikimedia.org/r/772450

Change 772450 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.1 refs T300203

https://gerrit.wikimedia.org/r/772450

Mentioned in SAL (#wikimedia-operations) [2022-03-21T18:04:37Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.1 refs T300203

brennen mentioned this in T304348: EntityContentDataCodec: InvalidArgumentException: Unsupported encoding format: text/plain.Mar 21 2022, 6:09 PM

brennen mentioned this in T304349: PHP Notice: Undefined index: taskType - causes Argument 1 passed to GrowthExperiments\NewcomerTasks\NewcomerTasksChangeTagsManager::apply() must be of the type string, null given.Mar 21 2022, 6:17 PM

Mentioned in SAL (#wikimedia-operations) [2022-03-21T18:19:36Z] <brennen> trainsperiment (T300203): 1.39.0-wmf.1 on all wikis; starting prep of wmf.2, will abort if needed

Change 772454 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772454

Change 772454 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772454

Mentioned in SAL (#wikimedia-operations) [2022-03-21T18:22:23Z] <brennen@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.2 refs T300203

brennen mentioned this in T304350: DisableOATHForUser: Error: Call to a member function getName() on null.Mar 21 2022, 6:37 PM

dduvall mentioned this in T304353: PHP Warning: preg_match() expects parameter 2 to be string, array given.Mar 21 2022, 7:10 PM

Mentioned in SAL (#wikimedia-operations) [2022-03-21T19:26:57Z] <brennen@deploy1002> Finished scap: testwikis wikis to 1.39.0-wmf.2 refs T300203 (duration: 64m 33s)

brennen mentioned this in T304356: UploadWizard.config.php: Error: Call to a member function asort() on null.Mar 21 2022, 7:34 PM

Change 772494 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772494

Change 772494 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772494

Mentioned in SAL (#wikimedia-operations) [2022-03-21T19:42:29Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.2 refs T300203

dduvall added a subtask: T304353: PHP Warning: preg_match() expects parameter 2 to be string, array given.Mar 21 2022, 8:06 PM

dduvall removed a subtask: T304353: PHP Warning: preg_match() expects parameter 2 to be string, array given.Mar 21 2022, 8:44 PM

Change 772501 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772501

Change 772501 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772501

Mentioned in SAL (#wikimedia-operations) [2022-03-21T20:49:04Z] <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.2 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-21T20:49:55Z] <dduvall@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.2 refs T300203 (duration: 00m 51s)

Change 772503 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772503

Change 772503 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.2 refs T300203

https://gerrit.wikimedia.org/r/772503

Mentioned in SAL (#wikimedia-operations) [2022-03-21T20:56:18Z] <dduvall@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.2 refs T300203

brennen mentioned this in T299583: Wikimedia\Assert\PreconditionException: Precondition failed: Range is not collapsed.Mar 21 2022, 9:13 PM

matmarex mentioned this in T304377: Wikimedia\Assert\PreconditionException: Precondition failed: Range is not collapsed (from CommentUtils::isSingleCommentSignedBy).Mar 21 2022, 10:35 PM

Change 772789 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772789

Change 772789 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772789

Mentioned in SAL (#wikimedia-operations) [2022-03-22T08:52:26Z] <jnuche@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.3 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-22T09:54:34Z] <jnuche@deploy1002> Finished scap: testwikis wikis to 1.39.0-wmf.3 refs T300203 (duration: 62m 07s)

Change 772794 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772794

Change 772794 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772794

Mentioned in SAL (#wikimedia-operations) [2022-03-22T10:03:37Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.3 refs T300203

The train is blocked by scap not flushing the PHP opcache on all the application servers. It only process 86 of them, should be a few hundred. T304414

Joe closed subtask T304414: scap no more restart php opcache on all servers as Resolved.Mar 22 2022, 1:08 PM

Change 772826 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772826

Change 772826 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772826

Mentioned in SAL (#wikimedia-operations) [2022-03-22T13:26:07Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.3 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-22T13:27:00Z] <jnuche@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.3 refs T300203 (duration: 00m 52s)

Change 772830 had a related patch set uploaded (by Jaime Nuche; author: Jaime Nuche):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772830

Change 772830 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/772830

Mentioned in SAL (#wikimedia-operations) [2022-03-22T13:39:55Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.3 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-22T17:25:07Z] <brennen> trainsperiment (T300203): with 1.39.0-wmf.3 on all wikis, we're paused for a planned catchup window - nothing to do at the moment, we'll deploy 1.39.0-wmf.4 tomorrow (2022-03-23).

Adding T304476 as a blocker for wmf4 (currently in wmf3).

• Jdlrobson removed a subtask: T304476: TypeError: insertLinkLinkTextField.getField(...).setValue(...).selectItem is not a function .Mar 23 2022, 2:18 PM

Mentioned in SAL (#wikimedia-operations) [2022-03-23T17:48:15Z] <brennen> trainsperiment (T300203): starting prep for 1.39.0-wmf.4

Change 773293 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773293

Change 773293 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773293

Mentioned in SAL (#wikimedia-operations) [2022-03-23T17:52:32Z] <brennen@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.4 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-23T18:42:13Z] <brennen@deploy1002> Finished scap: testwikis wikis to 1.39.0-wmf.4 refs T300203 (duration: 49m 41s)

Mentioned in SAL (#wikimedia-operations) [2022-03-23T18:47:35Z] <brennen> trainsperiment (T300203): 1.39.0-wmf.4 on testwikis; proceeding to groups 0-2 with 15 minute intervals for watching logs

Mentioned in SAL (#wikimedia-operations) [2022-03-23T18:53:07Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.4 refs T300203

Change 773304 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773304

Change 773304 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773304

Mentioned in SAL (#wikimedia-operations) [2022-03-23T18:59:21Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.4 refs T300203

Change 773326 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773326

Change 773326 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773326

Mentioned in SAL (#wikimedia-operations) [2022-03-23T19:08:14Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.4 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-23T19:09:07Z] <brennen@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.4 refs T300203 (duration: 00m 52s)

Change 773330 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773330

Change 773330 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773330

Mentioned in SAL (#wikimedia-operations) [2022-03-23T19:23:36Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.4 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-23T19:37:24Z] <brennen> trainsperiment (T300203): 1.39.0-wmf.4 on all wikis; logs seem clean - end of train deployment activities for the week, unless bugs emerge

brennen mentioned this in T304564: MWException: `[title]` is not a valid file title..Mar 23 2022, 8:56 PM

Mentioned in SAL (#wikimedia-operations) [2022-03-23T23:34:02Z] <brennen> trainsperiment (T300203): reverting to 1.39.0-wmf.3 on all wikis for T304564; will move forward again after a fix.

Change 773363 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/773363

Change 773363 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.3 refs T300203

https://gerrit.wikimedia.org/r/773363

Mentioned in SAL (#wikimedia-operations) [2022-03-23T23:38:02Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.3 refs T300203

brennen added a subtask: T304564: MWException: `[title]` is not a valid file title..Mar 23 2022, 11:38 PM

End-of-workday notes: See T304564: MWException: `[title]` is not a valid file title.; once that's patched the train can move forward again.

I think we're in a fairly good place. wmf.4 is only about a day's worth of delta from wmf.3, and has been on all wikis for most of the US afternoon.

T304564 fix has been backported and deployed

T304559 concerns RevisionStore I have poke platform engineering team about

Jdforrester-WMF closed subtask T304564: MWException: `[title]` is not a valid file title. as Resolved.Mar 24 2022, 1:03 PM

In T300203#7802754, @hashar wrote:

T304559 concerns RevisionStore I have poke platform engineering team about

Daniel and Lucas spoke about earlier today and given the rare occurrence we have decided it is not worth keeping it as a train blocker. T304559#7802768

Mentioned in SAL (#wikimedia-operations) [2022-03-24T16:13:27Z] <brennen> trainsperiment (T300203): blockers clear, logs triaged, rolling 1.39.0-wmf.4 out to all wikis again

Change 773579 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773579

Change 773579 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773579

Mentioned in SAL (#wikimedia-operations) [2022-03-24T16:20:33Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.4 refs T300203

Change 773582 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773582

Change 773582 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773582

Mentioned in SAL (#wikimedia-operations) [2022-03-24T16:24:17Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.4 refs T300203

Mentioned in SAL (#wikimedia-operations) [2022-03-24T16:25:25Z] <brennen@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.4 refs T300203 (duration: 01m 06s)

Change 773583 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773583

Change 773583 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.4 refs T300203

https://gerrit.wikimedia.org/r/773583

Mentioned in SAL (#wikimedia-operations) [2022-03-24T16:29:53Z] <brennen@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.4 refs T300203

brennen mentioned this in T304657: Kartographer: PHP Notice: Undefined property: stdClass::$type.Mar 24 2022, 8:37 PM

thcipriani closed this task as Resolved.Apr 1 2022, 12:19 AM

thcipriani claimed this task.

PleaseStand mentioned this in T310935: "A non well formed numeric value encountered" error when attempting to view T300203.Jun 18 2022, 3:35 AM

Dylsss removed Release Version.Jun 18 2022, 6:14 PM