Page MenuHomePhabricator
Create Task
Maniphest T352305

Multiple extensions' CI blocked again by cache corruption error in mwgate-node16-docker
Closed, DuplicatePublic
Actions

Description

Similar to T349986

Example output:

11:55:57 npm ERR! errno FETCH_ERROR
11:55:57 npm ERR! invalid json response body at https://registry.npmjs.org/zwitch reason: Invalid response body while trying to fetch https://registry.npmjs.org/zwitch: ENOENT: no such file or directory, stat '/cache/_cacache/content-v2/sha512/c4/13/4b9344a9b882cbfb7cb906102c2e604f34898d68810f1d66155b0e20d655acb63a5ff56f70e6e3ed49b8e9f386fe3ea02a636dd5a35d53ddaa35816fb787'

See https://integration.wikimedia.org/ci/job/mwgate-node16-docker/90347/console

Related Objects

Event Timeline

Jakob_WMDE created this task.Nov 29 2023, 3:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 29 2023, 3:29 PM
Lucas_Werkmeister_WMDE added subscribers: Jdforrester-WMF, Lucas_Werkmeister_WMDE.Nov 29 2023, 3:31 PM

CCing @Jdforrester-WMF who was already working on this in #wikimedia-releng.

Jdforrester-WMF renamed this task from Wikibase CI blocked again by cacache corruption error in mwgate-node16-docker to Multiple extensions' CI blocked again by cache corruption error in mwgate-node16-docker.Nov 29 2023, 3:32 PM
hashar added a project: Castor.Nov 29 2023, 3:38 PM

That is the same issue described on T295351 which to me is most probably a race condition specially showing up when there are a lot of patches being merged (and thus updating the cache on the fly). That is the case currently since for the last 3 hours CI has been hammered by a series of changes for T352284. See list at https://gerrit.wikimedia.org/r/q/topic:lsc-T352284

Lucas_Werkmeister_WMDE mentioned this in T352309: Gerrit maintenance bot should throttle uploading patches.Nov 29 2023, 3:50 PM
Stashbot added a comment.Nov 29 2023, 8:37 PM

Mentioned in SAL (#wikimedia-releng) [2023-11-29T20:37:41Z] <James_F> Ran jforrester@integration-castor05:~$ sudo rm -fR /srv/castor/castor-mw-ext-and-skins/master/mw*-node16-* for T352305

Urbanecm_WMF subscribed.Nov 29 2023, 8:39 PM
Jdforrester-WMF added a comment.Nov 29 2023, 11:19 PM

It was working briefly, but now it's corrupted again it seems. :-(

Silvan_WMDE subscribed.Nov 30 2023, 8:39 AM
Lucas_Werkmeister_WMDE added a comment.Dec 1 2023, 4:38 PM

I think it’s working better now? I haven’t encountered this error today (or, as far as I remember, yesterday).

Jdforrester-WMF added a comment.Dec 1 2023, 8:57 PM
In T352305#9375631, @Lucas_Werkmeister_WMDE wrote:

I think it’s working better now? I haven’t encountered this error today (or, as far as I remember, yesterday).

Yes, so far it seems to have stabilised as working, but it seems that it's just an unfortunate event away from recurring.

hashar closed this task as a duplicate of T295351: npm cache saved by castor get corrupted for unknown reason.Jan 22 2024, 8:43 AM
Jakob_WMDE mentioned this in T358312: Wikibase CI blocked by castor cache corruption issue.Feb 23 2024, 9:00 AM
Jakob_WMDE mentioned this in T370748: Wikibase CI failing due to npm cache corruption issue.Jul 23 2024, 8:19 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits