Risky Patch! 🚂🔥

• Change:
  • https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/827544
  • https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/827545
  • https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Elastica/+/827204
  • https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/824787
  • https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Translate/+/828050
• Summary:
  • Upgrades elasticsearch integration from 6.x to 7.x.
  • Elasticsearch 6 is running in the eqiad search cluster, elasticsearch 7 is running in the codfw search cluster.
  • wmf.27 and wmf.28 are write-compatible with both elasticsearch versions, but only read compatible with one version of elasticsearch.
  • The mediawiki-config patch (deployed prior to branch cut) sends all search query traffic in wmf.28 to codfw (elastic 7.x), while wmf.27 traffic will stay in eqiad (elastic 6.x).
  • Overall this effects several hundreds of requests per second that flow through mediawiki web, api and jobrunner servers.
• Test plan:
  • Monitor failure logging. Traffic levels are high enough that more edge cases will be uncovered by users than we could hope to manually verify.
• Places to monitor:
  • Logstash: mediawiki-errors
  • Grafana: mediawiki-errors
  • Grafana: Elasticsearch Percentiles - Failures / Rejections Section
• Revert plan: Rollback train
• Affected wikis:

all of them

• IRC contact: ebernhardson, dcausse
• UBN Task Projects/tags: Discovery-Search
• Would you like to backport this change rather than ride the train?: No

Probably worth considering the above as a blocker since there's a fix up.

Change 829875 had a related patch set uploaded (by TrainBranchBot; author: trainbranchbot):

[mediawiki/core@wmf/1.39.0-wmf.28] Branch commit for wmf/1.39.0-wmf.28

https://gerrit.wikimedia.org/r/829875

Change 829875 merged by jenkins-bot:

[mediawiki/core@wmf/1.39.0-wmf.28] Branch commit for wmf/1.39.0-wmf.28

https://gerrit.wikimedia.org/r/829875

Change 829879 had a related patch set uploaded (by TrainBranchBot; author: MediaWiki PreSync):

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.28

https://gerrit.wikimedia.org/r/829879

Change 829879 merged by jenkins-bot:

[operations/mediawiki-config@master] testwikis wikis to 1.39.0-wmf.28

https://gerrit.wikimedia.org/r/829879

Mentioned in SAL (#wikimedia-operations) [2022-09-06T03:02:32Z] <mwpresync@deploy1002> Started scap: testwikis wikis to 1.39.0-wmf.28 refs T314189

Mentioned in SAL (#wikimedia-operations) [2022-09-06T03:38:49Z] <mwpresync@deploy1002> Finished scap: testwikis wikis to 1.39.0-wmf.28 refs T314189 (duration: 36m 17s)

Change 830091 had a related patch set uploaded (by TrainBranchBot; author: Jaime Nuche):

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.28

https://gerrit.wikimedia.org/r/830091

Change 830091 merged by jenkins-bot:

[operations/mediawiki-config@master] group0 wikis to 1.39.0-wmf.28

https://gerrit.wikimedia.org/r/830091

Mentioned in SAL (#wikimedia-operations) [2022-09-06T08:09:25Z] <jnuche@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.39.0-wmf.28 refs T314189

We are getting 1.3M/day warnings for bad use of table name in building queries in RC. https://logstash.wikimedia.org/goto/e0e96107196477c7c3c1ca5854ee585e

It's probably something introduced before, or turned on this week but this needs fixing.

Change 830585 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@master] rdbms: Remove useless warning

https://gerrit.wikimedia.org/r/830585

I don't know if you'd consider T317187: GrowthExperiments Special:Homepage: investigate performance regression since September 6 2022 to be a deployment blocker. It would be helpful if other teams could say whether they're observing performance degradation for features they're monitoring, to understand if it's related to wmf.28 changes, or PHP 7.4 rollout, or multi-DC rollout work. The ElasticSearch patches (T314189#8200929) are responsible for part of the increased response times for Special:Homepage, as noted in T317187#8217380, and I don't think there's any way around that.

There is a general increase in every performance metric it seems: https://performance.wikimedia.org/#!/week

We probably need to look at flame graphs. php 7.4 usually shouldn't cause this, it's demonstrably faster.

In T314189#8217470, @Ladsgroup wrote:

There is a general increase in every performance metric it seems: https://performance.wikimedia.org/#!/week

We probably need to look at flame graphs. php 7.4 usually shouldn't cause this, it's demonstrably faster.

Blocking the train until we have some answers on this

In T314189#8218419, @jeena wrote:

In T314189#8217470, @Ladsgroup wrote:

There is a general increase in every performance metric it seems: https://performance.wikimedia.org/#!/week

We probably need to look at flame graphs. php 7.4 usually shouldn't cause this, it's demonstrably faster.

Blocking the train until we have some answers on this

Some options I can think of:

• rollback to group0 and group1 wikis to wmf.27, to help determine if there are code changes in core and/or extensions/skins that are contributing. The ElasticSearch changes are surely a contributor to what we see on Special:Homepage but shouldn't be responsble for the overall increase in performance metrics, AIUI
• rollback to php 7.2 for some period of time to observe what happens with https://performance.wikimedia.org/#!/week and other charts
• rollback to stage 3 of T279664: Progressive Multi-DC roll out, in order to rule out that as a contributing factor

In T314189#8218522, @kostajh wrote:

Some options I can think of:

• rollback to group0 and group1 wikis to wmf.27, to help determine if there are code changes in core and/or extensions/skins that are contributing. The ElasticSearch changes are surely a contributor to what we see on Special:Homepage but shouldn't be responsble for the overall increase in performance metrics, AIUI
• rollback to php 7.2 for some period of time to observe what happens with https://performance.wikimedia.org/#!/week and other charts
• rollback to stage 3 of T279664: Progressive Multi-DC roll out, in order to rule out that as a contributing factor

Just for some extra information, group1 wikis haven't been rolled forward yet and are still at wmf.27

wmf.27 broke all video transcoding as reported in T317069: TimedMediaHandler passes unrecognized option `-fpsmax` leading to transcoding failures.

In T314189#8219952, @TheDJ wrote:

wmf.27 broke all video transcoding as reported in T317069: TimedMediaHandler passes unrecognized option `-fpsmax` leading to transcoding failures.

Definitely due to -fpsmax being passed to FFmpeg by TimedMediaHandler but we run an older version. I don't think the issue should block this week train, we can most probably rollback the patch and deploy it to wmf.28 or get a hotfix.

I removed T317187: GrowthExperiments Special:Homepage: investigate performance regression since September 6 2022 as a blocker, see T317187#8220203.

In T314189#8217470, @Ladsgroup wrote:

There is a general increase in every performance metric it seems: https://performance.wikimedia.org/#!/week

We probably need to look at flame graphs. php 7.4 usually shouldn't cause this, it's demonstrably faster.

This afternoon I have investigated the bump in DomInteractive shown by that graph. Looking at navigation timing it seems it was between 7:40 and 10:00 on Sept 7th. That would match:

07:46 	<topranks> 	Depool eqsin from user traffic in advance of core router upgrades - T295690
09:48 	<topranks> 	Re-pooling eqsin for user traffic after successful core router upgrades - T295690

My theory is as we depooled eqsin, the traffic shifted to Europe / US West coast which increased the latency for people in Asia and around. If we have somewhere latency graphs per AS / region I would expect Asia to show a higher latency and everything else to haven't changed. So I think it can be dismissed.

A view for loadEventEnd at that time:

{F35513591 size=full}

Change 830926 had a related patch set uploaded (by TrainBranchBot; author: Jeena Huneidi):

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.28

https://gerrit.wikimedia.org/r/830926

Change 830926 merged by jenkins-bot:

[operations/mediawiki-config@master] group1 wikis to 1.39.0-wmf.28

https://gerrit.wikimedia.org/r/830926

Mentioned in SAL (#wikimedia-operations) [2022-09-08T19:11:25Z] <jhuneidi@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.39.0-wmf.28 refs T314189

Mentioned in SAL (#wikimedia-operations) [2022-09-08T19:15:05Z] <jhuneidi@deploy1002> Synchronized php: group1 wikis to 1.39.0-wmf.28 refs T314189 (duration: 03m 39s)

Change 830929 had a related patch set uploaded (by TrainBranchBot; author: Jeena Huneidi):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.28

https://gerrit.wikimedia.org/r/830929

Change 830929 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.28

https://gerrit.wikimedia.org/r/830929

Mentioned in SAL (#wikimedia-operations) [2022-09-08T19:36:24Z] <jhuneidi@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.28 refs T314189

Change 830935 had a related patch set uploaded (by TrainBranchBot; author: Jeena Huneidi):

[operations/mediawiki-config@master] group2 wikis to 1.39.0-wmf.27

https://gerrit.wikimedia.org/r/830935

Change 830935 merged by jenkins-bot:

[operations/mediawiki-config@master] group2 wikis to 1.39.0-wmf.27

https://gerrit.wikimedia.org/r/830935

Mentioned in SAL (#wikimedia-operations) [2022-09-08T19:56:19Z] <jhuneidi@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.39.0-wmf.27 refs T314189

Change 830944 had a related patch set uploaded (by TrainBranchBot; author: Jeena Huneidi):

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.28

https://gerrit.wikimedia.org/r/830944

Change 830944 merged by jenkins-bot:

[operations/mediawiki-config@master] all wikis to 1.39.0-wmf.28

https://gerrit.wikimedia.org/r/830944

Mentioned in SAL (#wikimedia-operations) [2022-09-08T21:08:46Z] <jhuneidi@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.39.0-wmf.28 refs T314189

Although wmf.28 is already out, the above linked task is for reverting / fixing a change that made the "@stable to override" IndexPager::makeLink unused without proper deprecation in this train. This broke Special:CheckUser's ability to page the results and will have broken any other code that overrided IndexPager::makeLink.

Change 831082 had a related patch set uploaded (by Krinkle; author: Amir Sarabadani):

[mediawiki/core@master] rdbms: Allow SubQuery objects in SelectQueryBuilder as table

https://gerrit.wikimedia.org/r/831082

Change 831082 merged by jenkins-bot:

[mediawiki/core@master] rdbms: Allow SubQuery objects in SelectQueryBuilder as table

https://gerrit.wikimedia.org/r/831082

Change 832322 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[mediawiki/core@wmf/1.40.0-wmf.1] rdbms: Allow SubQuery objects in SelectQueryBuilder as table

https://gerrit.wikimedia.org/r/832322

Change 830585 abandoned by Ladsgroup:

[mediawiki/core@master] rdbms: Remove useless warning

Reason:

Done in favor of Id9d6c1d0d4c40f43ab0d2

https://gerrit.wikimedia.org/r/830585

Change 832322 abandoned by Ladsgroup:

[mediawiki/core@wmf/1.40.0-wmf.1] rdbms: Allow SubQuery objects in SelectQueryBuilder as table

Reason:

Not needed

https://gerrit.wikimedia.org/r/832322