The manifest for docker-registry.discovery.wmnet/restricted/mediawiki-multiversion-debug:2025-03-27-200753-publish-81 references an image layer blob sha256:e7b2287766dc2a93ea9014f37470ba45fe8afcfb095221fd6ed3ed2db19c7c59:

...
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 1119805497,
         "digest": "sha256:e7b2287766dc2a93ea9014f37470ba45fe8afcfb095221fd6ed3ed2db19c7c59"
      },
...

The registry returns a blob of the right length but wrong hash:

$ curl -v -n https://docker-registry.wikimedia.org/v2/restricted/mediawiki-multiversion-debug/blobs/sha256:e7b2287766dc2a93ea9014f37470ba45fe8afcfb095221fd6ed3ed2db19c7c59 > sha256:e7b2287766dc2a93ea9014f37470ba45fe8afcfb095221fd6ed3ed2db19c7c59

...
< content-type: application/octet-stream
< content-length: 1119805497
< docker-content-digest: sha256:e7b2287766dc2a93ea9014f37470ba45fe8afcfb095221fd6ed3ed2db19c7c59
< docker-distribution-api-version: registry/2.0
< etag: "sha256:e7b2287766dc2a93ea9014f37470ba45fe8afcfb095221fd6ed3ed2db19c7c59"
...
< age: 0
< x-cache: cp1110 pass, cp1110 pass
< x-cache-status: pass
< server-timing: cache;desc="pass", host;desc="cp1110"
...

$ ls -l sha256:e7b2287766dc2a93ea9014f37470ba45fe8afcfb095221fd6ed3ed2db19c7c59
-rw-rw-r-- 1 dancy wikidev 1119805497 Mar 28 00:21 sha256:e7b2287766dc2a93ea9014f37470ba45fe8afcfb095221fd6ed3ed2db19c7c59

That is the expected size of the blob from the manifest.

$ sha256sum sha256:e7b2287766dc2a93ea9014f37470ba45fe8afcfb095221fd6ed3ed2db19c7c59
da9e1fa86230529d142f8385e1d13f8cbe308bf970a0a25607512de08f47ad29  

But the hash doesn't match.

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
role::docker_registry: re-enable the blob cache	Elukey	operations/puppet	production	+0 -3
role::docker_registry_had::registry: disable nginx cache	Elukey	operations/puppet	production	+3 -0
profile::docker_registry: turn off backend redirects for Swift	Elukey	operations/puppet	production	+1 -1
Rename docker_registry_ha's occurrences to docker_registry	Elukey	labs/private	master	+16 -16
Update Cumin alias for Docker registry	Muehlenhoff	operations/puppet	production	+1 -1
Remove old docker_registry_ha hiera keys	Alexandros Kosiaris	labs/private	master	+0 -18
docker_registry: Pass defaults to 2 option parameters	Alexandros Kosiaris	operations/puppet	production	+2 -2
docker_registry: Instantiate APUs s3 backend instance	Alexandros Kosiaris	operations/puppet	production	+31 -1
docker_registry: Make sure ports are in the right format	Alexandros Kosiaris	operations/puppet	production	+2 -2
docker_registry: Refactor to allow >1 instance	Alexandros Kosiaris	operations/puppet	production	+189 -99
docker_registry: Move rsyslog rules from init to web.pp	Alexandros Kosiaris	operations/puppet	production	+8 -9
docker_registry_ha: Refactor to make it docker_registry	Alexandros Kosiaris	operations/puppet	production	+120 -120
pontoon: Add stack registry	Alexandros Kosiaris	operations/puppet	production	+13 -0
registry: Add hiera for the new hierarchy	Alexandros Kosiaris	labs/private	master	+16 -1
registry: Minor Puppet cleanups	Alexandros Kosiaris	operations/puppet	production	+7 -12
role::deployment_server::kubernetes: limit Docker concurrent uploads	Elukey	operations/puppet	production	+1 -0
docker_registry_ha: remove debug logging	Elukey	operations/puppet	production	+0 -1
docker-registry: log nginx debug to separate file	Filippo Giunchedi	operations/puppet	production	+2 -1
docker_registry_ha: set debug logging for nginx	Elukey	operations/puppet	production	+1 -1
docker_registry_ha: split blob and auth cache	Elukey	operations/puppet	production	+33 -20

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
build-images.py: Don't sleep after full build in train-dev	repos/releng/release!189	dancy	main-Ia9831d65ecb031642f8cb47e672a6520eed581ce	main
make-container-image: sleep only after full mediawiki image build	repos/releng/release!165	swfrench	work/swfrench/T390251-pause-on-full-build	main
build-images.py: Temp sleep for swift consistency	repos/releng/release!164	cgoubert	T390251	main

Customize query in GitLab

Related Objects
Search...

Status	Subtype	Assigned	Task
Open	BUG REPORT	None	T390251 docker-registry.wikimedia.org keeps serving bad blobs
Resolved		• elukey	T394476 Onboard the Docker Registry to apus
Resolved		MatthewVernon	T417396 Upgrade apus' ceph to 18.2.7 (or .8 if already available)
Resolved		CDanis	T404742 Increase RAM and nginx tmpfs on docker registry hosts
Resolved		• elukey	T412951 Move the docker registry's /restricted prefix to Docker Distribution backed up by Ceph
Resolved		• elukey	T420978 Move the Docker Registry's /ml prefix to S3/apus
Open		None	T427175 Create new S3 backends for the Docker Registry service
Resolved		• elukey	T428022 Move all Machine Learning Docker images under the /ml prefix in the Docker Registry
Open		None	T432829 Move Docker images under the /v2/releng prefix to S3

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

In T390251#10771962, @elukey wrote:

I quickly checked the patches and the refactoring work is really nice, and I like the two new options. Really good job :)

I am a bit on the fence about how to deploy this in production, since we may encounter unexpected bugs for other use cases (like non-scap image push/pull etc..). Upstream will not review the patches since the swift backend is decommed, so we'll have to maintain our own fork for the time being. To safely test the new changes, it would be easy enough to spin up a new registry stack (VM + Redis instance + Swift bucket) and point scap and other tools to it incrementally, to quickly test if everything is good or not. But on the other hand, we are already planning to move away from Swift in favor of S3, so if we have to create a new stack I'd start with S3 directly.

The alternative is to build and deploy the new deb, deploy in prod and check how it goes during the next days, something that I am not totally fond of but it may be necessary (I am aware that Ahmon have probably tested multiple cases locally already, but the risk of unexpected side effects is not zero).

@akosiaris @Scott_French - lemme know if you have a preference, but this is a sign for me that we should start working on a medium term solution for the registry (even before talking about switching to something else). Maybe simply moving to apus/S3 could be enough?

We 've discussed this thing yesterday in the ServiceOps meeting. The 2 paths aren't particularly different, at least in the amount of work that is required to implement them. Whether we configure the /restricted namespace to go to an instance of the registry patched using @dancy's patches or a second instance of the same unpatched software using S3 as a backend, the configuration and testing side of the work needed is the same. What differs is:

the amount of work making sure APUs is up to the task for the S3 solution
the amount of work needed to build and package the software.

The latter is admittedly less, however it is also the one where we 'd be just putting a stopgap while also having to work on the former to put us on a sustainable path. So, we think we should go straight to the APUs/S3 solution, gambling that it will pay off more or less immediately, leaving the other approach as a hedge in case it doesn't.

Opened T394476 to see if apus can take over the current load that we have on Swift. After the sign-off we'll be able to reason about concrete next steps.

hashar unsubscribed.May 16 2025, 9:30 AM

Change #1154301 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] registry: Minor Puppet cleanups

https://gerrit.wikimedia.org/r/1154301

Change #1154302 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] docker_registry_ha: Refactor to make it docker_registry

https://gerrit.wikimedia.org/r/1154302

Change #1154301 merged by Alexandros Kosiaris:

[operations/puppet@production] registry: Minor Puppet cleanups

https://gerrit.wikimedia.org/r/1154301

Change #1155257 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] docker_registry: Move rsyslog rules from init to web.pp

https://gerrit.wikimedia.org/r/1155257

Change #1155258 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] docker_registry: Refactor to allow >1 instance

https://gerrit.wikimedia.org/r/1155258

Change #1155601 had a related patch set uploaded (by Elukey; author: Elukey):

[labs/private@master] Rename docker_registry_ha's occurrences to docker_registry

https://gerrit.wikimedia.org/r/1155601

Change #1156761 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[labs/private@master] registry: Add hiera for the new hierarchy

https://gerrit.wikimedia.org/r/1156761

Change #1156762 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[labs/private@master] Remove old docker_registry_ha hiera keys

https://gerrit.wikimedia.org/r/1156762

Change #1156761 merged by Alexandros Kosiaris:

[labs/private@master] registry: Add hiera for the new hierarchy

https://gerrit.wikimedia.org/r/1156761

Change #1156767 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] pontoon: Add stack registry

https://gerrit.wikimedia.org/r/1156767

Change #1156767 merged by Filippo Giunchedi:

[operations/puppet@production] pontoon: Add stack registry

https://gerrit.wikimedia.org/r/1156767

Change #1154302 merged by Alexandros Kosiaris:

[operations/puppet@production] docker_registry_ha: Refactor to make it docker_registry

https://gerrit.wikimedia.org/r/1154302

Change #1155257 merged by Alexandros Kosiaris:

[operations/puppet@production] docker_registry: Move rsyslog rules from init to web.pp

https://gerrit.wikimedia.org/r/1155257

Change #1155258 merged by Alexandros Kosiaris:

[operations/puppet@production] docker_registry: Refactor to allow >1 instance

https://gerrit.wikimedia.org/r/1155258

Mentioned in SAL (#wikimedia-operations) [2025-06-13T11:41:47Z] <akosiaris> T390251 re-enable puppet on registry1004 after merging puppet refactoring changes.

Change #1156809 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] docker_registry: Make sure ports are in the right format

https://gerrit.wikimedia.org/r/1156809

Change #1156809 merged by Alexandros Kosiaris:

[operations/puppet@production] docker_registry: Make sure ports are in the right format

https://gerrit.wikimedia.org/r/1156809

Mentioned in SAL (#wikimedia-operations) [2025-06-13T12:21:19Z] <akosiaris> T390251 re-enable puppet on all registries.

Change #1156829 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] docker_registry: Instantiate APUs s3 backend instance

https://gerrit.wikimedia.org/r/1156829

Change #1156835 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] docker_registry: Pass defaults to 2 option parameters

https://gerrit.wikimedia.org/r/1156835

Change #1156835 merged by Alexandros Kosiaris:

[operations/puppet@production] docker_registry: Pass defaults to 2 option parameters

https://gerrit.wikimedia.org/r/1156835

Change #1156829 merged by Alexandros Kosiaris:

[operations/puppet@production] docker_registry: Instantiate APUs s3 backend instance

https://gerrit.wikimedia.org/r/1156829

Change #1159291 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update Cumin alias for Docker registry

https://gerrit.wikimedia.org/r/1159291

Change #1156762 merged by Alexandros Kosiaris:

[labs/private@master] Remove old docker_registry_ha hiera keys

https://gerrit.wikimedia.org/r/1156762

Change #1159291 merged by Alexandros Kosiaris:

[operations/puppet@production] Update Cumin alias for Docker registry

https://gerrit.wikimedia.org/r/1159291

Change #1155601 abandoned by Alexandros Kosiaris:

[labs/private@master] Rename docker_registry_ha's occurrences to docker_registry

Reason:

I think https://gerrit.wikimedia.org/r/c/labs/private/+/1156761/ and https://gerrit.wikimedia.org/r/c/labs/private/+/1156762 cover this as well, I 'll abandon, but feel free to restore, if I missed something.

https://gerrit.wikimedia.org/r/1155601

• elukey mentioned this in T394778: Build and push images to the docker registry from ml-lab.Jul 4 2025, 12:26 PM

BTullis subscribed.Jul 4 2025, 1:42 PM

dancy opened https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/189

build-images.py: Don't sleep after full build in train-dev

dancy merged https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/189

build-images.py: Don't sleep after full build in train-dev

bd808 mentioned this in T401533: Why does build-and-push-container-images take 11 minutes?.Aug 9 2025, 5:47 PM

bd808 added a subtask: T394476: Onboard the Docker Registry to apus.Aug 12 2025, 10:52 PM

bd808 changed the subtype of this task from "Task" to "Bug Report".Aug 12 2025, 11:03 PM

@akosiaris looks like you had a lot of changes for making a new registry, is that registry ready? Or is it still in the testing phase?

In T390251#11084257, @thcipriani wrote:

@akosiaris looks like you had a lot of changes for making a new registry, is that registry ready? Or is it still in the testing phase?

The latter. The backend is setup and we 've had a couple of successful pushes, but also some weird behavior we want to reproduce and investigate. The plan is to switch just mediawiki's /restricted part to the new registry once we are confident

The /var/lib/nginx path is a separate mount point, using tmpfs, with a size of 4G.

Apparently this is too small for known work loads as today there was an issue pushing to the registry when this ran out of disk.

After the push worked at a later point this was back to 0 usage. (observed on registry2004)

16:45 < cdanis> 2025/09/16 15:56:55 [crit] 1731183#1731183: *1091214 pwrite() "/var/lib/nginx/body/0000003080" failed (28: No space left on device), client: 10.64.16.93, server: , request:  "PATCH /v2/restricted/mediawiki-multiversion/blobs/uploads....

Scott_French mentioned this in T404742: Increase RAM and nginx tmpfs on docker registry hosts.Sep 16 2025, 5:24 PM

CDanis closed subtask T404742: Increase RAM and nginx tmpfs on docker registry hosts as Resolved.Sep 16 2025, 6:58 PM

dancy mentioned this in T406392: failed to push docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.10.5-py3.11-2025-10-03-192132-3003d4328df66a0086a350fdd2ba1dbd80a235c5: unknown: blob upload invalid.Oct 3 2025, 9:16 PM

My understanding from the last updates is that we are not actively pushing anything to the new registry with apus as backend, we have just quickly tested some months ago. Is it the right understanding?

After T406392 I think that we really exhausted the amount of engineering hours that are tolerable to debug an issue, and we should really focus on forming a working group that moves us away from Swift. It is not an easy and quick task, so we may want to reconsider the option of patching the current registry's swift driver as interim solution (previous some testing of course).

Lemme know your thoughts! Happy to coordinate the efforts for a WG if needed.

per radosgw-admin user stats --uid=docker-registrythere are only 11 objects in that account, which I think equates to it not being currently used.

T406392 is a good reminder of the fact that the bandaids we might otherwise fall back on (e.g., sleeps, internal retries) are not available in all contexts, so even though we've largely focused on the MediaWiki image use case here, we really need a more systematic solution.

So yes, +1 to prioritizing either / both of migrating away from Swift (clearly a longer-term effort) and reviving Ahmon's driver improvements (also non-trivial to test / deploy, but still faster), though this will likely need to wait until the new year for any significant progress.

@akosiaris do you think that the idea of forming a dedicated working group for the next couple of quarters could be feasible? I can take care of kicking it off and finding volunteers (sounds like me and Scott are already in :D).

bking subscribed.Dec 15 2025, 6:29 PM

In T390251#11436436, @elukey wrote:

@akosiaris do you think that the idea of forming a dedicated working group for the next couple of quarters could be feasible? I can take care of kicking it off and finding volunteers (sounds like me and Scott are already in :D).

I don't see how else we can solve this without some dedicated and focused time on this one. It requires some discovery and testing and that takes time. Finding that time is the difficult part, I 'll raise this to management level and see how we can resource it.

Created T412951 as follow up task :)

BTullis mentioned this in T413080: Design and build the next generation of container-registry service for the WMF production realm.Dec 18 2025, 1:40 PM

Scott_French mentioned this in T412265: Pushing to the docker registry fails with 500 Internal Server Error.Dec 18 2025, 2:36 PM

• elukey added a subtask: T412951: Move the docker registry's /restricted prefix to Docker Distribution backed up by Ceph.Jan 9 2026, 11:17 AM

Change #1225526 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::docker_registry: turn off backend redirects for Swift

https://gerrit.wikimedia.org/r/1225526

Change #1225526 merged by Elukey:

[operations/puppet@production] profile::docker_registry: turn off backend redirects for Swift

https://gerrit.wikimedia.org/r/1225526

Mentioned in SAL (#wikimedia-operations) [2026-01-13T11:03:27Z] <elukey> disable HTTP redirects to the Swift backend for all the Docker registries - T390251

Mentioned in SAL (#wikimedia-operations) [2026-01-13T16:41:40Z] <elukey> roll restart docker-registry-swift daemons on registry* to pick up the new settings (apparently the service refresh issued by puppet didn't work as intended) - T390251

Scott_French merged a task: T412265: Pushing to the docker registry fails with 500 Internal Server Error.Jan 22 2026, 4:09 PM

Scott_French added subscribers: Urbanecm_WMF, jsn.sherman, EMcFarland-WMF and 4 others.

I've merged T412265: Pushing to the docker registry fails with 500 Internal Server Error into this task, as we believe it's another manifestation of the same class of failure modes discussed here.

One key point of note from the investigation on that task:

In T412265#11473181, we identified a correlation between certain swift operations (i.e., new ms-be hosts loading) that could plausibly lead to metadata "churn" in the cluster, thus also to more pronounced eventual consistency, and both the March 2025 period of issues captured originally here and the December 2025 period reported in T412265.

This is of course not a cause, but is a plausible trigger for the behavior we've seen. Our understanding of the cause remains the combination of eventual consistency in swift and poor design of the swift backend driver in the docker registry, which we intend to address by moving to S3-on-Ceph (T412951).

Blake subscribed.Jan 22 2026, 4:19 PM

Scott_French merged a task: T406392: failed to push docker-registry.discovery.wmnet/repos/data-engineering/airflow-dags:airflow-2.10.5-py3.11-2025-10-03-192132-3003d4328df66a0086a350fdd2ba1dbd80a235c5: unknown: blob upload invalid.Jan 23 2026, 12:56 AM

Scott_French added a subscriber: amastilovic.

I've now also merged T406392, for the same reason.

One key point of note from that task is that buildkit, as used in the Gitlab CI image build / push jobs, does not have the same retry behavior as we've seen with dockerd (see e.g., T406392#11252870).

Meaning, although retries can in some cases paper over eventual-consistency related issues for MediaWiki image pushes, we won't generally see that during push failures on the Gitlab CI side of things.

MLechvien-WMF edited projects, added: ServiceOps; removed: serviceops-deprecated.Feb 2 2026, 9:58 AM

MLechvien-WMF moved this task from Inbox to In Progress on the ServiceOps board.Feb 2 2026, 10:13 AM

akosiaris removed akosiaris as the assignee of this task.Feb 4 2026, 4:38 PM

MLechvien-WMF moved this task from In Progress to Radar (Awareness) on the ServiceOps board.Feb 9 2026, 5:20 PM

• elukey changed the status of subtask T412951: Move the docker registry's /restricted prefix to Docker Distribution backed up by Ceph from Open to Stalled.Feb 12 2026, 11:33 AM

• elukey changed the status of subtask T412951: Move the docker registry's /restricted prefix to Docker Distribution backed up by Ceph from Stalled to Open.Feb 25 2026, 9:39 AM

Mentioned in SAL (#wikimedia-operations) [2026-02-26T09:47:50Z] <elukey> move the Docker Registry's /v2/restricted (MediaWiki Docker image prefix) to s3/apus - T390251

dancy opened https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/236

make-container-image/build-images.py: Remove 5 minute sleep after full image build push

dancy merged https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/236

make-container-image/build-images.py: Remove 5 minute sleep after full image build push

dancy mentioned this in T413809: 1.46.0-wmf.18 deployment blockers.Mar 2 2026, 5:37 PM

• elukey closed subtask T394476: Onboard the Docker Registry to apus as Resolved.Mar 9 2026, 4:16 PM

• elukey closed subtask T412951: Move the docker registry's /restricted prefix to Docker Distribution backed up by Ceph as Resolved.

To keep archives happy - we solved the main painful problem, namely upload/push of MediaWiki docker images in T412951. The long term strategy is to move away from swift in favor of S3, but we should be mindful of the work tracked in T413080.

Maintenance_bot removed a project: Patch-For-Review.Mar 9 2026, 4:31 PM

Left a note in T413080#11688938 about a possible strategy.

• elukey added a subtask: T420978: Move the Docker Registry's /ml prefix to S3/apus.Mar 23 2026, 5:44 PM

JMeybohm merged a task: T422424: Another blob upload invalid error when pushing to docker-registry.May 21 2026, 9:42 AM

• elukey closed subtask T420978: Move the Docker Registry's /ml prefix to S3/apus as Resolved.May 25 2026, 7:16 AM

• elukey created subtask T427175: Create new S3 backends for the Docker Registry service.May 25 2026, 7:38 AM

The final and permanent fix should be done in T427175.

• elukey created subtask T428022: Move all Machine Learning Docker images under the /ml prefix in the Docker Registry.Jun 3 2026, 7:25 AM

• elukey removed a subtask: T428022: Move all Machine Learning Docker images under the /ml prefix in the Docker Registry.

Change #1304060 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::docker_registry: re-enable the blob cache

https://gerrit.wikimedia.org/r/1304060