Page MenuHomePhabricator

Wikibase and some other jobs become very slow 2026-09-03/04
Closed, ResolvedPublic

Assigned To
Authored By
Peter
Sep 4 2026, 3:59 AM
Referenced Files
F101821131: wikibase_npm_slow_resolution.png
Sep 4 2026, 4:17 PM
F101751603: mediawiki_node24.png
Sep 4 2026, 5:08 AM
F101750097: wikibase_browsertests.png
Sep 4 2026, 5:08 AM
F101740920: Screenshot 2026-09-04 at 05.44.17.png
Sep 4 2026, 3:59 AM

Description

Looking at the graphs today I can see that the CI time increased massively for Wikibase but also for some other projects:

Screenshot 2026-09-04 at 05.44.17.png (2,822×1,574 px, 563 KB)

Here's an example of a run:
https://integration.wikimedia.org/ci/job/quibble-with-Wikibase-extensions-browser-tests-only-vendor-php83/18851/console

npm --prefix extensions/Wikibase ci
00:01:01.707 INFO:quibble.commands:>>> Start: npm --prefix extensions/Wikibase ci
00:01:05.246 npm warn deprecated @humanwhocodes/config-array@0.13.0: Use @eslint/config-array instead
00:01:05.309 npm warn deprecated abab@2.0.6: Use your platform's native atob() and btoa() methods instead
00:01:05.449 npm warn deprecated @humanwhocodes/object-schema@2.0.3: Use @eslint/object-schema instead
00:01:05.528 npm warn deprecated domexception@4.0.0: Use your platform's native DOMException instead
00:01:06.037 npm warn deprecated node-domexception@1.0.0: Use your platform's native DOMException instead
00:01:06.583 npm warn deprecated glob@11.1.0: Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me
00:01:06.588 npm warn deprecated glob@10.5.0: Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me
00:01:06.588 npm warn deprecated glob@10.5.0: Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me
00:01:06.589 npm warn deprecated glob@10.5.0: Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me
00:01:06.589 npm warn deprecated glob@10.5.0: Old versions of glob are not supported, and contain widely publicized security vulnerabilities, which have been fixed in the current version. Please update. Support for old versions may be purchased (at exorbitant rates) by contacting i@izs.me
00:01:07.420 npm warn deprecated eslint@8.57.1: This version is no longer supported. Please see https://eslint.org/version-support for other options.
00:05:26.263 
00:05:26.263 > wikibase@0.1.0 postinstall
00:05:26.263 > npm-run-all -p install:*
00:05:26.263 
00:05:26.571 
00:05:26.571 > wikibase@0.1.0 install:reuse
00:05:26.571 > npm --prefix repo/domains/reuse ci
00:05:26.571 
00:05:26.601 
00:05:26.601 > wikibase@0.1.0 install:legacy-ui-serialization
00:05:26.601 > npm --prefix view/packages/wikibase-serialization install
00:05:26.601 
00:05:26.606 
00:05:26.606 > wikibase@0.1.0 install:rest-api
00:05:26.606 > npm --prefix repo/rest-api ci
00:05:26.606 
00:05:26.614 
00:05:26.614 > wikibase@0.1.0 install:legacy-ui-data-values
00:05:26.614 > npm --prefix view/packages/wikibase-data-values install
00:05:26.614 
00:05:26.615 
00:05:26.615 > wikibase@0.1.0 install:tainted-ref
00:05:26.615 > npm --prefix view/lib/wikibase-tainted-ref ci
00:05:26.615 
00:05:26.627 
00:05:26.627 > wikibase@0.1.0 install:legacy-ui-data-model
00:05:26.627 > npm --prefix view/packages/wikibase-data-model install
00:05:26.627 
00:05:26.640 
00:05:26.640 > wikibase@0.1.0 install:gql-explorer
00:05:26.640 > npm --prefix repo/domains/reuse/graphiql-explorer ci
00:05:26.640 
00:05:26.672 
00:05:26.672 > wikibase@0.1.0 install:legacy-ui-value-view
00:05:26.672 > npm --prefix view/packages/wikibase-data-values-value-view install
00:05:26.672

It seems like npm was very slow, I'm thinking that it could be the audit/fund flags that we disabled in Quibble for the selenium jobs that is causing it? For the selenium jobs there no increase in speed. For Wikibase and GrowthExperiment it is. But its would be a such coincidence that we just rolled out the fix for the selenium tests and then it broke for the rest of the CI?

Event Timeline

Looking at that specific job successful builds duration at https://grafana.wikimedia.org/d/ccaceff0-9ca6-4c89-9c4b-9138003439bf/zuul-job-prometheus?from=now-1d&to=now&timezone=utc&var-pipeline=$__all&var-job=quibble-with-Wikibase-extensions-browser-tests-only-vendor-php83&var-status=SUCCESS&viewPanel=panel-5 :

wikibase_browsertests.png (817×546 px, 41 KB)

The removal of audit/fund was part of Quibble 1.20.0 which I rolled on Sep 2nd at 15:40 UTC with https://gerrit.wikimedia.org/r/c/integration/config/+/1333894

The duration bump started at Sep 3rd 22:30 UTC.

It is not limited to that specific job though. mediawiki-node24 shows a similar pattern https://grafana.wikimedia.org/d/ccaceff0-9ca6-4c89-9c4b-9138003439bf/zuul-job-prometheus?from=now-1d&to=now&timezone=utc&var-pipeline=$__all&var-job=mediawiki-node24&var-status=SUCCESS&viewPanel=panel-5 (albeit it is triggered by Wikibase)

mediawiki_node24.png (827×335 px, 40 KB)

The bump starts at around the same time.

en the build time went from ~ 5 minutes up to ~ 12 minutes

I looked at two builds that happened on the same WMCS instance. On a normal build https://integration.wikimedia.org/ci/job/quibble-with-Wikibase-extensions-browser-tests-only-vendor-php83/18817/console

│  54.627s │ npm --prefix extensions/Wikibase ci                             │
│ 112.734s │ npm --prefix extensions/Wikibase run --if-present @selenium-    │
│          │ test                                                            │
│ 167.802s │ User commands: extensions/Wikibase/build/jenkins/mw-apply-wb-   │
│          │ settings.sh -r repo, cp --verbose                               │
│          │ $WORKSPACE/src/LocalSettings.php $LOG_DIR, php                  │
│          │ maintenance/update.php --quick, npm --prefix                    │
│          │ extensions/Wikibase ci, npm --prefix extensions/Wikibase run    │
│          │ --if-present @selenium-test                                     │

Versus https://integration.wikimedia.org/ci/job/quibble-with-Wikibase-extensions-browser-tests-only-vendor-php83/18863/console

│ 445.817s │ npm --prefix extensions/Wikibase ci                             │
│ 115.739s │ npm --prefix extensions/Wikibase run --if-present @selenium-    │
│          │ test                                                            │
│ 562.237s │ User commands: extensions/Wikibase/build/jenkins/mw-apply-wb-   │
│          │ settings.sh -r repo, cp --verbose                               │
│          │ $WORKSPACE/src/LocalSettings.php $LOG_DIR, php                  │
│          │ maintenance/update.php --quick, npm --prefix                    │
│          │ extensions/Wikibase ci, npm --prefix extensions/Wikibase run    │
│          │ --if-present @selenium-test                                     │

So yeah that is npm ci growing from 54 seconds up to 445 seconds.

I have looked at Wikibase repo or integration/config, and nothing has changed yesterday. I asked the WMCS team whether anything could have changed on their side and they have done no change yesterday (but maybe the link to internet has its own issues).

I'll try to reproduce / investigate further.

Looks like there is some issue with npmjs.org, see https://github.com/orgs/community/discussions/187552#discussioncomment-18280231

I can reproduce it on my laptop on a random JS repo, npm install is very slow, adding --no-auditmakes it fast.

We did roll npm ci --no-audit --no-fund within Quibble T435975 rQUIBBLE7cd822e27a21: npm: Skip audit and fund reports on install. That command is used by Quibble and would thus not be affected by the ongoing audit slowness.

However, Wikibase is a mono repository and shell out other npm ci installs:

git grep npm.*ci package.json 
package.json:           "install:bridge": "npm --prefix client/data-bridge ci",
package.json:           "install:tainted-ref": "npm --prefix view/lib/wikibase-tainted-ref ci",
package.json:           "install:rest-api": "npm --prefix repo/rest-api ci",
package.json:           "install:reuse": "npm --prefix repo/domains/reuse ci",
package.json:           "install:gql-explorer": "npm --prefix repo/domains/reuse/graphiql-explorer ci",

And those do not have --no-audit --no-fund. They are thus affected by the slowness (5 times due to 5 installs) which I guess explains the issue. Thank you @fnegri!!

We would need a global npm configuration (either via a npm config file or via environment variable if that is feasible). I'll comment on T435975 with the above infos.

Mentioned in SAL (#wikimedia-releng) [2026-09-04T14:43:32Z] <hashar> integration: set in Jenkins global config: NPM_CONFIG_AUDIT=false and NPM_CONFIG_FUND=false # T437008

Change #1335785 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: disable npm audit/fund/update-notifier via npmrc

https://gerrit.wikimedia.org/r/1335785

Change #1335785 merged by jenkins-bot:

[integration/config@master] dockerfiles: disable npm audit/fund/update-notifier via npmrc

https://gerrit.wikimedia.org/r/1335785

Change #1335924 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: add missing arg in COPY /usr/etc/npmrc

https://gerrit.wikimedia.org/r/1335924

Change #1335924 merged by Hashar:

[integration/config@master] dockerfiles: add missing arg in COPY /usr/etc/npmrc

https://gerrit.wikimedia.org/r/1335924

Mentioned in SAL (#wikimedia-releng) [2026-09-04T15:52:08Z] <hashar> integration: remove from Jenkins global config: NPM_CONFIG_AUDIT=false and NPM_CONFIG_FUND=false # T437008

Change #1335937 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: update node based jobs to disable audit/fund

https://gerrit.wikimedia.org/r/1335937

Change #1335937 merged by jenkins-bot:

[integration/config@master] jjb: update node based jobs to disable audit/fund

https://gerrit.wikimedia.org/r/1335937

hashar closed this task as Resolved.EditedSep 4 2026, 4:17 PM
hashar claimed this task.

From https://status.npmjs.org/incidents/l4f53bbr9200

Resolved
This incident has been resolved.
Posted 2 hours ago. Sep 04, 2026 - 13:48 UTC

Monitoring
A fix has been implemented and we are monitoring the results.
Posted 2 hours ago. Sep 04, 2026 - 13:27 UTC

Investigating
We are currently investigating this issue.
Posted 4 hours ago. Sep 04, 2026 - 11:29 UTC

And surely the jobs are fast again:

wikibase_npm_slow_resolution.png (520×318 px, 26 KB)

I'll have pushed and deployed the patches/updates to disable npm audit, fund (and update-notifier), so that is resolved once for good.

Huge thank you to @fnegri for having found the root cause and @Peter for having monitored before 6am! :party: