Mobileapps CI is broken because of docker error
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Jgiannelos
	May 5 2025, 1:20 PM

Description

Mobileapps CI is failing for non test-related reasons.
More specifically:

https://integration.wikimedia.org/ci/job/mobileapps-pipeline-test/777/console

+ docker container create --rm --attach STDOUT --attach STDERR --name plib-validate-huouh063 docker-registry.wikimedia.org/releng/ajv:latest --spec=draft2020 --errors=text -s schema.yaml -d config.yaml
Unable to find image 'docker-registry.wikimedia.org/releng/ajv:latest' locally
latest: Pulling from releng/ajv
080399fa9084: Already exists
fe23fed1e6be: Pulling fs layer
f4138ff92a20: Pulling fs layer
a3e04c63bf57: Pulling fs layer
835d8de5b27b: Pulling fs layer
f6af558a4599: Pulling fs layer
1233a6c85648: Pulling fs layer
313966963896: Pulling fs layer
c0aa6ecd50e5: Pulling fs layer
f6af558a4599: Waiting
1233a6c85648: Waiting
835d8de5b27b: Waiting
313966963896: Waiting
c0aa6ecd50e5: Waiting
fe23fed1e6be: Verifying Checksum
fe23fed1e6be: Download complete
f4138ff92a20: Verifying Checksum
f4138ff92a20: Download complete
f6af558a4599: Verifying Checksum
f6af558a4599: Download complete
835d8de5b27b: Verifying Checksum
835d8de5b27b: Download complete
313966963896: Verifying Checksum
313966963896: Download complete
c0aa6ecd50e5: Verifying Checksum
c0aa6ecd50e5: Download complete
a3e04c63bf57: Verifying Checksum
a3e04c63bf57: Download complete
1233a6c85648: Verifying Checksum
1233a6c85648: Download complete
failed to register layer: open /srv/docker/overlay2/18a3eefd03265675dd1e97c725b3fd0bb8c8bd1fa4e7163e93755327e335a41f/committed: no such file or directory

Event Timeline

Jgiannelos created this task.May 5 2025, 1:20 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 5 2025, 1:20 PM

I have cleaned up /srv/docker/overlay2 this morning :(

hashar added a project: ci-test-error.May 5 2025, 1:43 PM

Mentioned in SAL (#wikimedia-releng) [2025-05-05T13:50:38Z] <hashar> contint1002 and contint2002: deleted /srv/docker/image/overlay2 following the deletion of /srv/docker/overlay2 earlier today # T393373

I have deleted the layers earlier today (/srv/docker/overlay2) but the database in /srv/docker/image/overlay2 was still referring to them which laid to the error. I have deleted that second directory on both hosts and confirmed I could pull the image:

contint2002$ sudo docker pull docker-registry.wikimedia.org/releng/ajv:latest
...
Digest: sha256:a3ec9a03b62d64ce77e91b2cc6670206f8c533b8e22b06b142eab970dbc13c75
Status: Downloaded newer image for docker-registry.wikimedia.org/releng/ajv:latest
docker-registry.wikimedia.org/releng/ajv:latest

contint1002$ sudo docker pull docker-registry.wikimedia.org/releng/ajv:latest
...
Digest: sha256:a3ec9a03b62d64ce77e91b2cc6670206f8c533b8e22b06b142eab970dbc13c75
Status: Downloaded newer image for docker-registry.wikimedia.org/releng/ajv:latest
docker-registry.wikimedia.org/releng/ajv:latest

Thanks I just sent a recheck

Now I am getting:

#4 docker-image://docker-registry.wikimedia.org/wikimedia/blubber-buildkit:v0.11.1@sha256:e9b2163f0d25c31efa5607922c5f1a46453812e0d6a010d64aa9c2799f29a2d0
#4 sha256:0384c1da5df35ebb3e25d997ea59c4f04d1aa87a080a61b9a8334611d72c2284
#4 CACHED
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: extract requires an applier

Jgiannelos reopened this task as Open.May 5 2025, 2:06 PM

Eeek :)

Mentioned in SAL (#wikimedia-releng) [2025-05-05T14:32:42Z] <hashar> contint1002 and contint2002: deleted /srv/docker/buildkit following the deletion of /srv/docker/overlay2 earlier today # T393373

I have deleted yet another directory (/srv/docker/buildkit), did a recheck on https://gerrit.wikimedia.org/r/c/mediawiki/services/mobileapps/+/1141878 and this time it passed! Sorry for the mess @Jgiannelos and thank you for having reported it.

thcipriani added a project: Essential-Work.May 9 2025, 12:18 AM

Mobileapps CI is broken because of docker error Closed, ResolvedPublicActions

Description

Event Timeline

Mobileapps CI is broken because of docker error
Closed, ResolvedPublic
Actions