Wikibase CI failing due to npm cache corruption issue
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Jakob_WMDE
	Jul 23 2024, 8:19 AM

Description

npm ERR! path /cache/_cacache/content-v2/sha512/65/9d/07d365d9a1cdde2aecf1e8e522d65a2f55b2845349805f0b2c13b6f26792c6805ab04dfdeacae752681a2eb2e8bb35b57c32150b4a3c018ddbcaaaf9c6eb
npm ERR! errno ENOENT
npm ERR! enoent Invalid response body while trying to fetch https://registry.npmjs.org/y18n: ENOENT: no such file or directory, stat '/cache/_cacache/content-v2/sha512/65/9d/07d365d9a1cdde2aecf1e8e522d65a2f55b2845349805f0b2c13b6f26792c6805ab04dfdeacae752681a2eb2e8bb35b57c32150b4a3c018ddbcaaaf9c6eb'

https://integration.wikimedia.org/ci/job/mwgate-node18/43108/consoleText

Similar to T358312, T349986, T352305, ...

Related Objects

Mentioned Here: T370766: Add WMDE staff working on development of Wikibase software
T349986: invalid json response body at https://registry.npmjs.org/vue reason: Invalid response body while trying to fetch https://registry.npmjs.org/vue: ENOENT: no such file or directory
T352305: Multiple extensions' CI blocked again by cache corruption error in mwgate-node16-docker
T358312: Wikibase CI blocked by castor cache corruption issue

Event Timeline

Jakob_WMDE created this task.Jul 23 2024, 8:19 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 23 2024, 8:19 AM

Lucas_Werkmeister_WMDE edited projects, added ci-test-error (WMF-deployed Build Failure), Wikidata; removed ci-test-error.Jul 23 2024, 8:34 AM

Muhammad_Yasser_Jazirahly_WMDE subscribed.Jul 23 2024, 8:50 AM

WMDE-leszek subscribed.Jul 23 2024, 9:02 AM

hashar claimed this task.Jul 23 2024, 10:21 AM

hashar triaged this task as Unbreak Now! priority.

Hi, I have deleted the faulty cache.

The process is to look up in the Jenkins job console the subpath that is used to restore/store the cache:

Defined: CASTOR_NAMESPACE="castor-mw-ext-and-skins/master/mwgate-node18"

Then on delete it from Castor:

ssh integration-castor05.integration.eqiad1.wikimedia.cloud \
  sudo rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwgate-node18

I guess we should add at least a few WMDE people to the integration WMCS project with the sudo permission so that you can easily nuke faulty caches.

Thanks @hashar, I've requested the addition in T370766.

Do you think it makes sense to add those instructions to https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Troubleshooting? (or some other page)
I'm basically not clear how well established that page is as a "CI troubleshooting cheatsheet" - if one existed, we'd surely be advertising it here at WMDE once our staff gets the permissions for those machines.

• Jayve20 changed the task status from Resolved to Declined.Jul 27 2024, 6:48 AM

• Jayve20 removed hashar as the assignee of this task.

• Jayve20 removed projects: Wikidata, ci-test-error (WMF-deployed Build Failure), Castor, Release-Engineering-Team.

• Jayve20 removed subscribers: WMDE-leszek, Aklapper, Jdforrester-WMF and 2 others.

Aklapper changed the task status from Declined to Resolved.Jul 27 2024, 2:00 PM

Aklapper assigned this task to hashar.

Aklapper added projects: Wikidata, ci-test-error (WMF-deployed Build Failure), Castor, Release-Engineering-Team.

Aklapper added subscribers: WMDE-leszek, Jdforrester-WMF, hashar and 2 others.