Change #1054406 had a related patch set uploaded (by TrainBranchBot; author: trainbranchbot):

[mediawiki/core@wmf/1.43.0-wmf.14] Branch commit for wmf/1.43.0-wmf.14

https://gerrit.wikimedia.org/r/1054406

Change #1054406 merged by jenkins-bot:

[mediawiki/core@wmf/1.43.0-wmf.14] Branch commit for wmf/1.43.0-wmf.14

https://gerrit.wikimedia.org/r/1054406

Change #1054410 had a related patch set uploaded (by TrainBranchBot; author: MediaWiki PreSync):

[operations/mediawiki-config@master] testwikis wikis to 1.43.0-wmf.14

https://gerrit.wikimedia.org/r/1054410

Change #1054410 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.43.0-wmf.14

https://gerrit.wikimedia.org/r/1054410

Mentioned in SAL (#wikimedia-operations) [2024-07-16T03:03:00Z] <mwpresync@deploy1002> Started scap sync-world: testwikis wikis to 1.43.0-wmf.14 refs T366959

Mentioned in SAL (#wikimedia-operations) [2024-07-16T03:53:56Z] <mwpresync@deploy1002> Finished scap: testwikis wikis to 1.43.0-wmf.14 refs T366959 (duration: 50m 56s)

Change #1054630 had a related patch set uploaded (by TrainBranchBot; author: Ahmon Dancy):

[operations/mediawiki-config@master] group0 wikis to 1.43.0-wmf.14

https://gerrit.wikimedia.org/r/1054630

Change #1054630 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.43.0-wmf.14

https://gerrit.wikimedia.org/r/1054630

Mentioned in SAL (#wikimedia-operations) [2024-07-16T18:14:10Z] <dancy@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.43.0-wmf.14 refs T366959

Change #1054922 had a related patch set uploaded (by TrainBranchBot; author: Ahmon Dancy):

[operations/mediawiki-config@master] group1 wikis to 1.43.0-wmf.14

https://gerrit.wikimedia.org/r/1054922

Change #1054922 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.43.0-wmf.14

https://gerrit.wikimedia.org/r/1054922

Mentioned in SAL (#wikimedia-operations) [2024-07-17T18:10:12Z] <dancy@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.43.0-wmf.14 refs T366959

Change #1054926 had a related patch set uploaded (by TrainBranchBot; author: Ahmon Dancy):

[operations/mediawiki-config@master] group0 wikis to 1.43.0-wmf.14

https://gerrit.wikimedia.org/r/1054926

Change #1054926 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.43.0-wmf.14

https://gerrit.wikimedia.org/r/1054926

Mentioned in SAL (#wikimedia-operations) [2024-07-17T18:46:06Z] <dancy@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.43.0-wmf.14 refs T366959

I rolled back to group0 by request of @Marostegui who noted a massive increase in s1 activity.

A few minutes after the deployment quite a number of s1 replicas started to show lag - the master started to have contention too.
https://grafana.wikimedia.org/goto/qcuiQWuSg?orgId=1

That's weird, s1 is enwiki and that's group2. I don't know anything that would do a cross wiki read to enwiki from a group1 wiki not to mention writes. Do we have some samples on what was overwhelming s1?

In T366959#9993226, @Ladsgroup wrote:

That's weird, s1 is enwiki and that's group2. I don't know anything that would do a cross wiki read to enwiki from a group1 wiki not to mention writes. Do we have some samples on what was overwhelming s1?

It was very weird indeed but the time sort of matched the deployment. A few minutes after it, the master stared to build up contention (which makes sense as it's not an immediate thing), then it recovered a bit and then it increased again (to a lower level).

Most of slaves started to show lag, which means there was some heavy writers coming in or the master was too loaded.

The safest option was to rollback and then decide, that's why I requested it. It was very late in the EU evening and we didn't seem to have anyone else around from the SRE team to support another deployment to see if it was just a bad coincidence or something inflicted by the code.

@ DBA's: Explicitly asking: Are we fine deploying 1.43.0-wmf.14 to group1 wikis again (lag spike was coincidence time-wise), or are we not (train is blocked as some code change is needed first)? Thanks!

My understanding of comments in T370304 and of the IRC backlog in #mediawiki_security is that the train can proceed.

Change #1055148 had a related patch set uploaded (by TrainBranchBot; author: Andre Klapper):

[operations/mediawiki-config@master] group1 wikis to 1.43.0-wmf.14

https://gerrit.wikimedia.org/r/1055148

Change #1055148 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.43.0-wmf.14

https://gerrit.wikimedia.org/r/1055148

Mentioned in SAL (#wikimedia-operations) [2024-07-18T08:13:40Z] <aklapper@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.43.0-wmf.14 refs T366959

T370396: Error: Call to a member function inNamespace() on null seems to be a new error in this train, FYI. Not sure if it should be a train blocker.

Thanks. I've seen that issue but all occurrences so far are on old 1.43.0-wmf.13 (thus not a train blocker) and limited to a single wiki.

Yeah, I just noticed the wrong train version too, sorry for the noise.

In T366959#9993586, @Aklapper wrote:

My understanding of comments in T370304 and of the IRC backlog in #mediawiki_security is that the train can proceed.

Yes, it is fine to continue

Can you skip arwiki from this Update. Arabic month names changed and this may stop some bots from run there

That's not how train deployments work. If there's a specific patch you don't want to roll out to all wikis you could potentially revert and backport that one patch. But we won't keep one wiki on a legacy version because unspecified bots are slow (that was whoever wrote the patch's responsibility before putting it on the train).

In T366959#9995482, @Pppery wrote:

That's not how train deployments work. If there's a specific patch you don't want to roll out to all wikis you could potentially revert and backport that one patch. But we won't keep one wiki on a legacy version because unspecified bots are slow (that was whoever wrote the patch's responsibility before putting it on the train).

Can tody changes in Arabic translation included in the version?

Any changes you made on translatewiki today won't get deployed until next train. And backporting i18n changes is apparently difficult and not generally done. So I think you're out of luck.

(I have no authority here, just describing how the process works in my understanding)

In T366959#9995535, @hubaishan wrote:

In T366959#9995482, @Pppery wrote:

That's not how train deployments work. If there's a specific patch you don't want to roll out to all wikis you could potentially revert and backport that one patch. But we won't keep one wiki on a legacy version because unspecified bots are slow (that was whoever wrote the patch's responsibility before putting it on the train).

Can tody changes in Arabic translation included in the version?

What are the specific translation commits that are needed? Backporting translations is more time-consuming than other backports and we try to avoid them. But if its a small number of changes and it saves arwiki work, we could look into it.

The specific changes being complained about seem to be https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1054194/1/languages/i18n/ar.json

They have already been reverted on translatewiki.

Hi we just discovered a train blocker that impacts dark mode. We're creating a bug and working on a patch now but please don't roll the train forward until we've given the green light!

Splitting the Arabic month issue out to T370456

Change #1055293 had a related patch set uploaded (by TrainBranchBot; author: Ahmon Dancy):

[operations/mediawiki-config@master] group2 to 1.43.0-wmf.14

https://gerrit.wikimedia.org/r/1055293

Change #1055293 merged by jenkins-bot:

[operations/mediawiki-config@master] group2 to 1.43.0-wmf.14

https://gerrit.wikimedia.org/r/1055293

Mentioned in SAL (#wikimedia-operations) [2024-07-18T21:01:31Z] <dancy@deploy1002> rebuilt and synchronized wikiversions files: group2 to 1.43.0-wmf.14 refs T366959