Page MenuHomePhabricator
Create Task
Maniphest T348243

Wikibase CI is broken
Closed, ResolvedPublic
Actions

Description

https://integration.wikimedia.org/ci/job/mwgate-node16-docker/74682/console

11:58:50 npm ERR! code ENOENT
11:58:53 npm ERR! syscall stat
11:58:53 npm ERR! path /cache/_cacache/content-v2/sha512/e0/2a/cc4f90ce1d30e4e076039739b0bb815509c21823c81abaf79ab2b742bb620a205389ddfd2fffc3031890832fda03da9a73fc0bf56f4363df8e47f4cf9720
11:58:53 npm ERR! errno -2
11:58:53 npm ERR! enoent ENOENT: no such file or directory, stat '/cache/_cacache/content-v2/sha512/e0/2a/cc4f90ce1d30e4e076039739b0bb815509c21823c81abaf79ab2b742bb620a205389ddfd2fffc3031890832fda03da9a73fc0bf56f4363df8e47f4cf9720'
11:58:53 npm ERR! enoent This is related to npm not being able to find a file.
11:58:53 npm ERR! enoent

Details

SubjectRepoBranchLines +/-
DNM: Empty change to test CImediawiki/extensions/Wikibasemaster+0 -0
Customize query in gerrit

Related Objects

Event Timeline

Muhammad_Yasser_Jazirahly_WMDE created this task.Oct 5 2023, 12:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 5 2023, 12:44 PM
Lucas_Werkmeister_WMDE subscribed.Oct 5 2023, 12:45 PM

There are also a lot of warnings earlier (copied from another build):

npm WARN tarball tarball data for magic-string@0.25.7 (sha512-4CrMT5DOHTDk4HYDlzmwu4FVCcIYI8gauveasrdCu2IKIFOJ3f0v/8MDGJCDL9oD2ppz/Av1b0Nj345H9M+XIA==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for istanbul-lib-instrument@5.1.0 (sha512-czwUz525rkOFDJxfKK6mYfIs9zBKILyrZQxjz3ABhjQXhbhFsSbo1HW/BFcsDnfJYJWA6thRR5/TUY2qs5W99Q==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for vfile@https://registry.npmjs.org/vfile/-/vfile-4.2.1.tgz (sha512-O6AE4OskCG5S1emQ/4gl8zK586RqA3srz3nfK/Viy0UPToBc5Trp9BVFb1u0CjsKrAWwnpr4ifM/KBXPWwJbCA==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for zwitch@https://registry.npmjs.org/zwitch/-/zwitch-1.0.5.tgz (sha512-V50KMwwzqJV0NpZIZFwfOD5/lyny3WlSzRiXgA0G7VUnRlqttta1L6UQIHzd6EuBY/cHGfwTIck7w1yH6Q5zUw==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for ws@https://registry.npmjs.org/ws/-/ws-7.5.5.tgz (sha512-BAkMFcAzl8as1G/hArkxOxq3G7pjUqQ3gzYbLL0/5zNkph70e+lCoxBGnm6AW1+/aiNeV4fnKqZ8m4GZewmH2w==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for wordwrap@https://registry.npmjs.org/wordwrap/-/wordwrap-1.0.0.tgz (sha1-J1hIEIkUVqQXHI0CJkQa3pDLyus=) seems to be corrupted. Trying again.
npm WARN tarball tarball data for webpack-dev-server@https://registry.npmjs.org/webpack-dev-server/-/webpack-dev-server-3.11.2.tgz (sha512-A80BkuHRQfCiNtGBS1EMf2ChTUs0x+B3wGDFmOeT4rmJOHhHTCH2naNxIHhmkr0/UillP4U3yeIyv1pNp+QDLQ==) seems to be corrupted. Trying again.
npm WARN tarball tarball data for walker@https://registry.npmjs.org/walker/-/walker-1.0.7.tgz (sha1-L3+bj9ENZ3JisYqITijRlhjgKPs=) seems to be corrupted. Trying again.
npm WARN tarball tarball data for webpack@https://registry.npmjs.org/webpack/-/webpack-4.46.0.tgz (sha512-6jJuJjg8znb/xRItk7bkT0+Q7AHCYjjFnvKIWQPkNIOyRqoCGvkOs0ipeQzrqz4l5FtN5ZI/ukEHroeX/o1/5Q==) seems to be corrupted. Trying again.
Lucas_Werkmeister_WMDE added a project: ci-test-error (WMF-deployed Build Failure).Oct 5 2023, 12:46 PM
Silvan_WMDE subscribed.Oct 5 2023, 1:30 PM
hashar subscribed.Oct 5 2023, 1:45 PM

That is the npm cache used by CI which is corrupted somehow, possibly due to a race condition. I have never figured it out.

At the top of the job output, an environment variable is defined which sets under which directory of the central cache to save artifacts:

09:57:51 Defined: CASTOR_NAMESPACE="castor-mw-ext-and-skins/master/mwgate-node16-docker"

And I then deleted it entirely:

ssh integration-castor05.integration.eqiad1.wikimedia.cloud \
    sudo rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwgate-node16-docker

That should fix the build failure on mwgate-node16-docker.

Lucas_Werkmeister_WMDE added a comment.Oct 5 2023, 2:20 PM

Seems to be working again, thanks!

Lucas_Werkmeister_WMDE closed this task as Resolved.Oct 5 2023, 2:36 PM
Lucas_Werkmeister_WMDE assigned this task to hashar.
Silvan_WMDE reopened this task as Open.Oct 26 2023, 7:21 AM

Re-opening, since this is happening again.

Example builds:
https://integration.wikimedia.org/ci/job/mwext-node16-rundoc-docker/16615/console
https://integration.wikimedia.org/ci/job/mwext-node16-rundoc-docker/16671/console
https://integration.wikimedia.org/ci/job/mwext-node16-rundoc-docker/16672/console

gerritbot added a comment.Oct 26 2023, 7:22 AM

Change 968947 had a related patch set uploaded (by Silvan Heintze; author: Silvan Heintze):

[mediawiki/extensions/Wikibase@master] DNM: Empty change to test CI

https://gerrit.wikimedia.org/r/968947

gerritbot added a project: Patch-For-Review.Oct 26 2023, 7:22 AM
Stashbot added a comment.Oct 26 2023, 7:38 AM

Mentioned in SAL (#wikimedia-releng) [2023-10-26T07:38:39Z] <hashar> integrastion-castor05: sudo rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-node16-rundoc-docker # T348243

hashar closed this task as Resolved.Oct 26 2023, 8:00 AM

@Silvan_WMDE I have cleared the cache for this job/branch from the central instance.

Silvan_WMDE added a comment.Oct 26 2023, 8:09 AM

Thank you, that fixed it. 👍️

gerritbot added a comment.Oct 26 2023, 8:09 AM

Change 968947 abandoned by Silvan Heintze:

[mediawiki/extensions/Wikibase@master] DNM: Empty change to test CI

Reason:

fixed

https://gerrit.wikimedia.org/r/968947

Maintenance_bot removed a project: Patch-For-Review.Oct 26 2023, 8:10 AM
Jakob_WMDE subscribed.Oct 30 2023, 8:47 AM

@hashar this error keeps coming back :/

08:26:09 npm ERR! code FETCH_ERROR
08:26:09 npm ERR! errno FETCH_ERROR
08:26:09 npm ERR! invalid json response body at https://registry.npmjs.org/webpack reason: Invalid response body while trying to fetch https://registry.npmjs.org/webpack: ENOENT: no such file or directory, stat '/cache/_cacache/content-v2/sha512/64/56/2942ff80fee9219cb8dce63b958ad87611bc4f61a7e89d73d7480e024ff1f812bcf0f07edf0725f4d306fa3b885381b0f5abac217d9199805d7dbb855eab'

https://integration.wikimedia.org/ci/job/mwgate-node16-docker/82190/consoleFull
https://integration.wikimedia.org/ci/job/mwgate-node16-docker/82201/consoleFull
https://integration.wikimedia.org/ci/job/mwgate-node16-docker/82202/consoleFull

Lucas_Werkmeister_WMDE added a comment.Oct 30 2023, 11:49 AM

(I think this latest instance was fixed in T349986.)

Jakob_WMDE reopened this task as Open.Oct 30 2023, 3:45 PM

@hashar thanks for fixing this earlier. Unfortunately it looks like the error is back already. Is there anything we can do on our side to avoid this?

https://integration.wikimedia.org/ci/job/mwgate-node16-docker/82365/console
https://integration.wikimedia.org/ci/job/mwgate-node16-docker/82353/console

16:21:22 npm ERR! code FETCH_ERROR
16:21:22 npm ERR! errno FETCH_ERROR
16:21:22 npm ERR! invalid json response body at https://registry.npmjs.org/@jest%2fglobals reason: Invalid response body while trying to fetch https://registry.npmjs.org/@jest%2fglobals: ENOENT: no such file or directory, stat '/cache/_cacache/content-v2/sha512/af/6c/d25ae24369232ca5c876220afe1135a42e14aa4625c447cce8db449268a008f2f2f7f1b508763c7253b7a887ebff3de72b6a7052b48dd2902f4b284b6cf0'
Jakob_WMDE closed this task as Resolved.Oct 31 2023, 8:58 AM

It's working again now.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL