Page MenuHomePhabricator
Create Task
Maniphest T342338

toolforge docker registry: unable to pull docker-registry.tools.wmflabs.org/toolforge-distroless-base:latest
Closed, ResolvedPublic
Actions

Description

Getting:

$ docker pull docker-registry.tools.wmflabs.org/toolforge-distroless-base:latest
latest: Pulling from toolforge-distroless-base
dd5ad9c9c29f: Retrying in 1 second 
960043b8858c: Retrying in 1 second 
b4ca4c215f48: Downloading  599.6kB/599.6kB
eebb06941f3e: Waiting 
02cd68c0cbf6: Waiting 
d3c894b5b2b0: Waiting 
b40161cd83fc: Waiting 
46ba3f23f1d3: Waiting 
4fa131a1b726: Waiting 
01f38fc88b34: Waiting 
6e24d515f042: Waiting 
0460cb7a0f85: Waiting 
4a2d7bc46f58: Waiting 
unexpected EOF
$ docker pull docker-registry.tools.wmflabs.org/toolforge-distroless-base:latest
latest: Pulling from toolforge-distroless-base
dd5ad9c9c29f: Pulling fs layer 
960043b8858c: Pulling fs layer 
b4ca4c215f48: Downloading 
eebb06941f3e: Waiting 
02cd68c0cbf6: Waiting 
d3c894b5b2b0: Waiting 
b40161cd83fc: Waiting 
46ba3f23f1d3: Waiting 
4fa131a1b726: Waiting 
01f38fc88b34: Waiting 
6e24d515f042: Waiting 
0460cb7a0f85: Waiting 
4a2d7bc46f58: Waiting 
unknown blob

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
toolforge: buildservice: upload_images_to_repo: add -debug suffix to distrolesscloud/wmcs-cookbooksmain+3 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

• aborrero created this task.Jul 20 2023, 11:49 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 20 2023, 11:49 AM
• aborrero moved this task from Backlog to Doing on the User-aborrero board.Jul 20 2023, 1:17 PM
• aborrero mentioned this in T321188: [tbs] fix/improve the updating of the buildpack/tekton images in the local repo.
Stashbot added a comment.Jul 20 2023, 2:34 PM

Mentioned in SAL (#wikimedia-cloud) [2023-07-20T14:34:06Z] <arturo> deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 again with newer image (T342338, T321188)

gerritbot added a comment.Jul 20 2023, 2:54 PM

Change 940175 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/wmcs-cookbooks@main] toolforge: buildservice: upload_images_to_repo: add -debug suffix to distroless

https://gerrit.wikimedia.org/r/940175

gerritbot added a project: Patch-For-Review.Jul 20 2023, 2:54 PM
Stashbot added a comment.Jul 20 2023, 3:19 PM

Mentioned in SAL (#wikimedia-cloud) [2023-07-20T15:19:41Z] <arturo> deploying https://gitlab.wikimedia.org/repos/cloud/toolforge/buildservice/-/merge_requests/6 again with newer image (T342338, T321188)

gerritbot added a comment.Jul 20 2023, 3:23 PM

Change 940175 merged by Arturo Borrero Gonzalez:

[cloud/wmcs-cookbooks@main] toolforge: buildservice: upload_images_to_repo: add -debug suffix to distroless

https://gerrit.wikimedia.org/r/940175

• aborrero mentioned this in rCCKBb5e82b46b223: toolforge: buildservice: upload_images_to_repo: add -debug suffix to distroless.Jul 20 2023, 3:24 PM
• aborrero closed this task as Resolved.Jul 20 2023, 3:28 PM

I pushed a new image with a -debug suffix which apparently fixed the broken one.

The theory by @taavi is that the several operations to bring back the old image caused some intermediary layer to get lost.
Pushing a new image (even with different name) resulted in the intermediary layers getting right again.

Note to self: we should treat the docker registry as an append-only DB and not do rollbacks or data rewrites/rescues as it really leaves the registry in inconsistent state.

Maintenance_bot removed a project: Patch-For-Review.Jul 20 2023, 3:30 PM
• aborrero added a comment.Jul 20 2023, 3:52 PM

See also: https://lists.wikimedia.org/hyperkitty/list/cloud-admin@lists.wikimedia.org/thread/RLRWGRP5H3R2NEHBIZPGOJJ7W5KGEXLY/

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits