Page MenuHomePhabricator
Create Task
Maniphest T437697

Incomplete package-lock.json in mediawiki/core makes npm rewrite 2500 cache files on every CI build
Open, Needs TriagePublic
Actions

Description

When pushing https://gerrit.wikimedia.org/r/c/integration/config/+/1338086 I could see that we still have many files that gets copied to Castor every time in CI.

It looks like the problem comes from that Core's package-lock.json has 1962 entries and 1145 of them have no resolved and no integrity field. All are dev dependencies. That seems to have come from the change https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1276810

I think without those fields the npm cache can not be used directly. In CI this means every Quibble job that installs core's npm dependencies, which includes all the standard core and extension gate jobs, rewrites about 2,500 npm cache files per build, and castor copies all of them to Castor on each save.

I think the easiest way is to remove the lock file and node_modules and run npm install inside of node fresh. Or @Mhurd
what do you think?

I also tried to recreate how this could have happened I think this happens if you remove the lock file but keep node_modules and run a new install.

We also should check all gated extensions and verify that those lock files look ok.

package-lock.json that lack the resolved and integrity fields can be updated using:

npx --package=package-lock-fix -- package-lock-fix package-lock.json

Then reindenting the file from 2 spaces to tabulations.

Details

Other Assignee
Mhurd
Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
build: package-lock.json: drop src/mobile.startupHasharmediawiki/extensions/MobileFrontendREL1_45+0 -3
utils: script to verify package-lock.json integrityHasharintegration/configmaster+146 -0
bridge: regenerate package-lock.jsonHasharmediawiki/extensions/WikibaseREL1_43+7 K -2 K
build: regenerate package-lock.jsonHasharmediawiki/extensions/CodeMirrorREL1_45+1 K -571
build: regenerate package-lock.jsonHasharmediawiki/extensions/MobileFrontendREL1_46+2 K -867
build: add package-lock.jsonHasharmediawiki/extensions/JsonFormsmaster+6 K -0
build: regenerate package-lock.jsonHasharmediawiki/extensions/PopupsREL1_45+2 K -811
selenium: drop redundant package.json type=moduleHasharmediawiki/extensions/VisualEditormaster+0 -3
build: regenerate package-lock.jsonHasharmediawiki/extensions/PopupsREL1_46+2 K -710
build: regenerate package-lock.jsonHasharmediawiki/extensions/Popupsmaster+1 K -662
build: regenerate package-lock.jsonHasharmediawiki/extensions/CodeMirrorREL1_43+2 K -908
build: add package-lock.jsonHasharmediawiki/extensions/MediaWikiFarmREL1_46+1 K -0
build: add package-lock.jsonHasharmediawiki/extensions/MediaWikiFarmREL1_45+1 K -0
build: add package-lock.jsonHasharmediawiki/extensions/MediaWikiFarmREL1_43+1 K -0
build: add package-lock.jsonHasharmediawiki/extensions/MediaWikiFarmmaster+1 K -0
tests: add structure test validate package-lock.jsonHasharmediawiki/coremaster+77 -0
build: regenerate package-lock.jsonHasharmediawiki/extensions/ChartREL1_43+14 -0
build: Regenerate package-lock.jsonPhedenskogmediawiki/coremaster+4 K -5 K
build: Regenerate package-lock.jsonPhedenskogmediawiki/extensions/MobileFrontendmaster+2 K -1 K
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Peter created this task.Fri, Sep 11, 10:50 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFri, Sep 11, 10:50 AM
Peter renamed this task from Incomplete package-lock.json in mediawiki/core makes npm rewrite 2,500 cache files on every CI build to Incomplete package-lock.json in mediawiki/core makes npm rewrite 2500 cache files on every CI build.Fri, Sep 11, 10:51 AM
gerritbot added a comment.Fri, Sep 11, 11:06 AM

Change #1339693 had a related patch set uploaded (by Phedenskog; author: Phedenskog):

[mediawiki/core@master] build: Regenerate package-lock.json

https://gerrit.wikimedia.org/r/1339693

gerritbot added a project: Patch-For-Review.Fri, Sep 11, 11:06 AM
Peter claimed this task.Fri, Sep 11, 11:55 AM
Peter added a parent task: T437452: Get the gate-and-submit pipeline feedback loop p75 for core below 10 minute.
Peter updated Other Assignee, added: Mhurd.
Peter moved this task from Backlog to In Review on the Test Platform (Arusha 30) board.Mon, Sep 14, 5:10 AM
gerritbot added a comment.Wed, Sep 16, 4:25 PM

Change #1342305 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/core@master] tests: add structure test validate package-lock.json

https://gerrit.wikimedia.org/r/1342305

hashar added a comment.EditedWed, Sep 16, 7:20 PM

I wrote a small script:

#!/usr/bin/env python

import json
import sys

for f in sys.argv[1:]:
    lock = json.load(open(f))

    match lock['lockfileVersion']:
        case 1 | 2:
            deps_key = 'dependencies'
        case 3:
            deps_key = 'packages'
        case _:
            print("%s: unrecognized lockfileVersion!" % f)
    try:
        for name, package in lock[deps_key].items():
            if name == "":
                continue
            if 'resolved' in package and 'integrity' in package:
                continue
            print('%s: package %s misses integrity or resolved' % (f, name))
            break

    except Exception as e:
        print("Got exception in %s: %s" % (f, e))
        continue

Invoked using: cd ~/projects/mediawiki && find {extensions,skins} -maxdepth 2 -name package-lock.json | xargs ./checkpackagejson and surely it found some others affected repositories.

extensions/MobileFrontend/package-lock.json
extensions/Wikibase/view/packages/wikibase-data-values-value-view/package-lock.json
extensions/Wikibase/view/lib/wikibase-data-values-value-view/package-lock.json
extensions/Wikibase/client/data-bridge/package-lock.json
extensions/PersonalDashboard/package-lock.json
extensions/Popups/package-lock.json
extensions/ModeratorToolkit/package-lock.json
hashar added subscribers: Peter.Hedenskog, Jdforrester-WMF.Wed, Sep 16, 7:26 PM

@Peter.Hedenskog you have made a patch which fixes mediawiki/core and I think can be merged as-is.

I commented there with some extra steps to prevent a regression in the future. From my comment on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1339693/comments/d52da665_2f73fd93 :

I'll write a structure test that check the package-lock.json and would fail when a package misses those keys, that would prevent a regression.

That is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1342305 , which will certainly break CI for a few extensions that have a wrong package-lock.json file so we should fix them first. We will certainly want to backport it.

verify all extensions & skins across supported branches

Done above T437697#12330550. Note this does not cover other repositories that have package-lock.json.

we certainly need a guide somewhere to explain how to regenerate the package-lock.json which the failing test can point to

I have no idea how developers do it, we will want it documented I guess. Maybe @Jdforrester-WMF would know?

gerritbot added a comment.Thu, Sep 17, 12:46 PM

Change #1342667 had a related patch set uploaded (by Phedenskog; author: Phedenskog):

[mediawiki/extensions/MobileFrontend@master] build: Regenerate package-lock.json

https://gerrit.wikimedia.org/r/1342667

gerritbot added a comment.Thu, Sep 17, 2:44 PM

Change #1339693 merged by jenkins-bot:

[mediawiki/core@master] build: Regenerate package-lock.json

https://gerrit.wikimedia.org/r/1339693

ReleaseTaggerBot added a project: MW-1.47-notes (1.47.0-wmf.21; 2026-09-22).Thu, Sep 17, 3:00 PM
Peter moved this task from In Review to Finishing Touches on the Test Platform (Arusha 30) board.Thu, Sep 17, 5:15 PM
gerritbot added a comment.Thu, Sep 17, 11:00 PM

Change #1342667 merged by jenkins-bot:

[mediawiki/extensions/MobileFrontend@master] build: Regenerate package-lock.json

https://gerrit.wikimedia.org/r/1342667

Peter moved this task from Finishing Touches to Done on the Test Platform (Arusha 30) board.Fri, Sep 18, 4:11 AM
hashar mentioned this in T427450: Find the root cause for the long Castor wait times.Fri, Sep 18, 1:04 PM
hashar mentioned this in T353925: gate-and-submit backlogged due to waiting for castor-save-workspace-cache.
hashar added a parent task: T427450: Find the root cause for the long Castor wait times.
Peter closed this task as Resolved.Fri, Sep 18, 1:51 PM
gerritbot added a comment.Mon, Sep 21, 1:34 PM

Change #1343551 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] utils: script to verify package-lock.json integrity

https://gerrit.wikimedia.org/r/1343551

hashar reopened this task as Open.Mon, Sep 21, 1:36 PM

Reopening since there are a few repositories that are missing package-lock.json as indicated earlier at T437697#12330550 . I went ahead and uploaded an enhanced version of the script as https://gerrit.wikimedia.org/r/c/integration/config/+/1343551 , it now check all MediaWiki supported branches (with the exception of wmf/* ones).

~/projects/integration/config/utils/checkpackagejson core vendor skins/* extensions/*
extensions/Chart:REL1_43: package node_modules/ansi-styles misses integrity or resolved
extensions/CodeMirror:REL1_43: package node_modules/@aashutoshrathi/word-wrap misses integrity or resolved
extensions/CodeMirror:REL1_45: package node_modules/@aashutoshrathi/word-wrap misses integrity or resolved
extensions/DataMaps:REL1_46: MISSING package-lock.json
extensions/DataMaps:master: MISSING package-lock.json
extensions/JsonForms:master: MISSING package-lock.json
extensions/MediaWikiFarm:REL1_43: MISSING package-lock.json
extensions/MediaWikiFarm:REL1_45: MISSING package-lock.json
extensions/MediaWikiFarm:REL1_46: MISSING package-lock.json
extensions/MediaWikiFarm:master: MISSING package-lock.json
extensions/MobileFrontend:REL1_45: package node_modules/mobile.startup misses integrity or resolved
extensions/MobileFrontend:REL1_46: package node_modules/@ampproject/remapping misses integrity or resolved
extensions/MobileFrontend:master: package node_modules/mobile.returnToApp misses integrity or resolved
extensions/ModeratorToolkit:master: package node_modules/vitest-plugin-mediawiki misses integrity or resolved
extensions/PersonalDashboard:REL1_46: package node_modules/vitest-plugin-mediawiki misses integrity or resolved
extensions/PersonalDashboard:master: package node_modules/vitest-plugin-mediawiki misses integrity or resolved
extensions/Popups:REL1_45: package node_modules/@aashutoshrathi/word-wrap misses integrity or resolved
extensions/Popups:REL1_46: package node_modules/@aashutoshrathi/word-wrap misses integrity or resolved
extensions/Popups:master: package node_modules/@aashutoshrathi/word-wrap misses integrity or resolved
extensions/QuickSurveys:master-backup: MISSING package-lock.json
gerritbot added a comment.Mon, Sep 21, 1:41 PM

Change #1343553 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/Chart@REL1_43] build: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1343553

gerritbot added a comment.Mon, Sep 21, 1:59 PM

Change #1343555 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/CodeMirror@REL1_43] build: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1343555

gerritbot added a comment.Mon, Sep 21, 2:15 PM

Change #1343553 merged by Jforrester:

[mediawiki/extensions/Chart@REL1_43] build: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1343553

gerritbot added a comment.Mon, Sep 21, 2:46 PM

Change #1343570 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/CodeMirror@REL1_45] build: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1343570

gerritbot added a comment.Mon, Sep 21, 3:04 PM

Change #1343575 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/JsonForms@master] build: add package-lock.json

https://gerrit.wikimedia.org/r/1343575

AMarkossyan-WMF closed this task as Resolved.Mon, Sep 21, 3:59 PM
gerritbot added a comment.Mon, Sep 21, 3:59 PM

Change #1343584 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/Popups@master] build: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1343584

gerritbot added a comment.Mon, Sep 21, 4:02 PM

Change #1343588 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/Popups@REL1_46] build: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1343588

hashar reopened this task as Open.Mon, Sep 21, 4:04 PM
hashar added a subscriber: AMarkossyan-WMF.

@AMarkossyan-WMF it is not resolved yet :) We need to update a few more extensions which have the same problem and add a system to prevent the issue from happening again.

gerritbot added a comment.Mon, Sep 21, 4:05 PM

Change #1343589 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/Popups@REL1_45] build: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1343589

gerritbot added a comment.Mon, Sep 21, 4:12 PM

Change #1343591 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/MediaWikiFarm@master] build: add package-lock.json

https://gerrit.wikimedia.org/r/1343591

gerritbot added a comment.Mon, Sep 21, 4:14 PM

Change #1343593 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/MediaWikiFarm@REL1_46] build: add package-lock.json

https://gerrit.wikimedia.org/r/1343593

gerritbot added a comment.Mon, Sep 21, 4:18 PM

Change #1343594 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/MediaWikiFarm@REL1_46] build: add package-lock.json

https://gerrit.wikimedia.org/r/1343594

gerritbot added a comment.Mon, Sep 21, 4:18 PM

Change #1343595 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/MediaWikiFarm@REL1_45] build: add package-lock.json

https://gerrit.wikimedia.org/r/1343595

gerritbot added a comment.Mon, Sep 21, 4:18 PM

Change #1343596 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/MediaWikiFarm@REL1_43] build: add package-lock.json

https://gerrit.wikimedia.org/r/1343596

gerritbot added a comment.Mon, Sep 21, 4:27 PM

Change #1343591 merged by Hashar:

[mediawiki/extensions/MediaWikiFarm@master] build: add package-lock.json

https://gerrit.wikimedia.org/r/1343591

gerritbot added a comment.Mon, Sep 21, 4:28 PM

Change #1343596 merged by Hashar:

[mediawiki/extensions/MediaWikiFarm@REL1_43] build: add package-lock.json

https://gerrit.wikimedia.org/r/1343596

gerritbot added a comment.Mon, Sep 21, 4:28 PM

Change #1343595 merged by Hashar:

[mediawiki/extensions/MediaWikiFarm@REL1_45] build: add package-lock.json

https://gerrit.wikimedia.org/r/1343595

gerritbot added a comment.Mon, Sep 21, 4:29 PM

Change #1343594 merged by Hashar:

[mediawiki/extensions/MediaWikiFarm@REL1_46] build: add package-lock.json

https://gerrit.wikimedia.org/r/1343594

hashar mentioned this in rEMWF443ebed93136: build: add package-lock.json.Mon, Sep 21, 5:33 PM
hashar mentioned this in rEMWFe59eaede1216: build: add package-lock.json.
hashar mentioned this in rEMWF1b0f702a8915: build: add package-lock.json.
hashar mentioned this in rEMWF93fd37de3903: build: add package-lock.json.
hashar updated the task description. (Show Details)Mon, Sep 21, 8:01 PM
hashar updated the task description. (Show Details)Mon, Sep 21, 8:05 PM
gerritbot added a comment.Tue, Sep 22, 5:41 AM

Change #1343840 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/VisualEditor@master] selenium: drop redundant package.json type=module

https://gerrit.wikimedia.org/r/1343840

gerritbot added a comment.Tue, Sep 22, 10:30 PM

Change #1343555 merged by Jforrester:

[mediawiki/extensions/CodeMirror@REL1_43] build: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1343555

gerritbot added a comment.Tue, Sep 22, 10:32 PM

Change #1343584 merged by jenkins-bot:

[mediawiki/extensions/Popups@master] build: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1343584

gerritbot added a comment.Wed, Sep 23, 12:38 AM

Change #1343588 merged by jenkins-bot:

[mediawiki/extensions/Popups@REL1_46] build: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1343588

gerritbot added a comment.Wed, Sep 23, 7:49 AM

Change #1343840 abandoned by Hashar:

[mediawiki/extensions/VisualEditor@master] selenium: drop redundant package.json type=module

Reason:

Per https://gerrit.wikimedia.org/r/c/mediawiki/extensions/VisualEditor/+/1343840/comments/fe012efa_e24c0472 , it is needed otherwise NodeJS emits a warning and overall inferring whether a file is a Common or ES module takes causes some overhead.

https://gerrit.wikimedia.org/r/1343840

gerritbot added a comment.Wed, Sep 23, 11:50 AM

Change #1343589 merged by Hashar:

[mediawiki/extensions/Popups@REL1_45] build: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1343589

gerritbot added a comment.Wed, Sep 23, 1:32 PM

Change #1344276 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/MobileFrontend@REL1_45] build: package-lock.json: drop src/mobile.startup

https://gerrit.wikimedia.org/r/1344276

gerritbot added a comment.Wed, Sep 23, 8:46 PM

Change #1343575 merged by Umherirrender:

[mediawiki/extensions/JsonForms@master] build: add package-lock.json

https://gerrit.wikimedia.org/r/1343575

AMarkossyan-WMF edited projects, added: Test Platform (San Miguel de Allende 31); removed: Test Platform (Arusha 30).Thu, Sep 24, 10:33 AM
AMarkossyan-WMF moved this task from Backlog to Done on the Test Platform (San Miguel de Allende 31) board.
Peter added a comment.Fri, Sep 25, 5:34 AM

Here's the impact this fix had on the total wait time in the queue for the castor save:

packagelock.png (3,288×1,064 px, 282 KB)

Peter removed a subscriber: Peter.Hedenskog.Fri, Sep 25, 5:38 AM
gerritbot added a comment.Mon, Sep 28, 12:52 PM

Change #1345539 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/MobileFrontend@REL1_46] build: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1345539

AMarkossyan-WMF added a comment.Mon, Sep 28, 12:54 PM
In T437697#12346240, @hashar wrote:

@AMarkossyan-WMF it is not resolved yet :) We need to update a few more extensions which have the same problem and add a system to prevent the issue from happening again.

oops, I misunderstood then! Thanks for opening the ticket again :)

gerritbot added a comment.Mon, Sep 28, 1:32 PM

Change #1345552 had a related patch set uploaded (by Hashar; author: Hashar):

[mediawiki/extensions/Wikibase@REL1_43] bridge: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1345552

gerritbot added a comment.Mon, Sep 28, 2:59 PM

Change #1345539 merged by jenkins-bot:

[mediawiki/extensions/MobileFrontend@REL1_46] build: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1345539

gerritbot added a comment.Mon, Sep 28, 3:09 PM

Change #1343570 merged by jenkins-bot:

[mediawiki/extensions/CodeMirror@REL1_45] build: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1343570

gerritbot added a comment.Mon, Sep 28, 3:29 PM

Change #1345552 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@REL1_43] bridge: regenerate package-lock.json

https://gerrit.wikimedia.org/r/1345552

gerritbot added a comment.Sat, Oct 3, 7:28 PM

Change #1344276 merged by jenkins-bot:

[mediawiki/extensions/MobileFrontend@REL1_45] build: package-lock.json: drop src/mobile.startup

https://gerrit.wikimedia.org/r/1344276

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits