Page MenuHomePhabricator

25% latency regression July 2nd due to InstallerExtensionSelector service running in production
Closed, ResolvedPublic

Description

https://grafana.wikimedia.org/d/000000066/resourceloader?panelId=45&fullscreen&orgId=1&from=1593203378189&to=1595383471212

Screenshot 2020-07-22 at 23.46.21.png (1×2 px, 372 KB)

https://performance.wikimedia.org/arclamp/svgs/daily/2020-07-21.excimer.load.svgz

This shows a huge block that wasn't there before:

export.png (1×2 px, 294 KB)

Related code is from InstallerExtensionSelector.php, which was introduced last week and seems to be scanning unlimited and without batching or caching and then parsing extension.json files. This is presumably not meant to be enabled in production :)

Event Timeline

Legoktm added a subscriber: Legoktm.

This was introduced in https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/46eabe275c923ca7827435a36d6576150cea8899

I briefly discussed this with Krinkle on IRC that a full revert makes sense, because of the performance degradation and the other problems I noticed while reviewing the patch that I'll explain in a bit.

(Correction: The regression takes up 20% of today's latency; which means, 100/80=, it is a 25% latency regression.)

Krinkle renamed this task from 20% latency regression July 2nd due to InstallerExtensionSelector service running in production to 25% latency regression July 2nd due to InstallerExtensionSelector service running in production.Jul 22 2020, 10:52 PM

Change 615584 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/core@master] Revert "Add a new type of database to the installer from extension"

https://gerrit.wikimedia.org/r/615584

Change 615439 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/core@wmf/1.36.0-wmf.1] Revert "Add a new type of database to the installer from extension"

https://gerrit.wikimedia.org/r/615439

Change 615440 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/core@wmf/1.35.0-wmf.41] Revert "Add a new type of database to the installer from extension"

https://gerrit.wikimedia.org/r/615440

Change 615584 merged by jenkins-bot:
[mediawiki/core@master] Revert "Add a new type of database to the installer from extension"

https://gerrit.wikimedia.org/r/615584

Change 615439 merged by jenkins-bot:
[mediawiki/core@wmf/1.36.0-wmf.1] Revert "Add a new type of database to the installer from extension"

https://gerrit.wikimedia.org/r/615439

Change 615440 merged by jenkins-bot:
[mediawiki/core@wmf/1.35.0-wmf.41] Revert "Add a new type of database to the installer from extension"

https://gerrit.wikimedia.org/r/615440

Mentioned in SAL (#wikimedia-operations) [2020-07-23T00:16:56Z] <legoktm@deploy1001> Synchronized php-1.36.0-wmf.1/includes/: T258664: Revert "Add a new type of database to the installer from extension" (duration: 01m 09s)

Mentioned in SAL (#wikimedia-operations) [2020-07-23T00:22:47Z] <legoktm@deploy1001> Synchronized php-1.35.0-wmf.41/includes/libs/rdbms/database/Database.php: T258664: Revert "Add a new type of database to the installer from extension" (duration: 01m 05s)

Mentioned in SAL (#wikimedia-operations) [2020-07-23T00:24:13Z] <legoktm@deploy1001> Synchronized php-1.35.0-wmf.41/includes/: T258664: Revert "Add a new type of database to the installer from extension" (2/2) (duration: 01m 08s)

Change 615441 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/core@REL1_35] Revert "Add a new type of database to the installer from extension"

https://gerrit.wikimedia.org/r/615441

Change 615441 merged by jenkins-bot:
[mediawiki/core@REL1_35] Revert "Add a new type of database to the installer from extension"

https://gerrit.wikimedia.org/r/615441

Looks like it's dropping now... I'll let @Krinkle verify and close.

Screenshot_2020-07-22 ResourceLoader - Grafana.png (951×1 px, 166 KB)

Thanks @Krinkle @Legoktm

@CCicalese_WMF let's review with the rest of the team next week when you're back to figure out how the regression got all the way out.

AMooney closed this task as Resolved.EditedJul 28 2020, 8:47 PM
AMooney added a subscriber: AMooney.

The code has been reverted. Please reopen if there is more work to be done.