18 and 98 minutes.

Unimpressive. I wonder what it's doing that takes so much time? How many images exist in each bucket?

My idea is to pick either releng or restricted and start pruning old images via https://wikitech.wikimedia.org/wiki/Docker-registry#Deleting_images

In the long term, for the restricted images case: There exists a weekly scap clean-images systemd timer that runs on deployment.eqiad.wmnet. It has an image expiration policy for deleting local images. Ideally it would be extended to request corresponding deletions from the registry.

In the meantime, restricted images over 3 weeks old can surely be deleted.

... and then run GC during a special maintenance (human driven with the Registry in read-only) when the S3 buckets cross a certain threshold. Having it fully automated is the goal, but I'd go for an interim solution for the moment.

Agreed.

Very high level breakdown for /restricted:

elukey@registry1004:~$ curl http://localhost:5002/v2/_catalog
{"repositories":["restricted/mediawiki-multiversion","restricted/mediawiki-multiversion-cli","restricted/mediawiki-multiversion-debug","restricted/mediawiki-singleversion","restricted/mediawiki-singleversion-cli","restricted/mediawiki-singleversion-debug","restricted/mediawiki-webserver"]}
elukey@registry1004:~$ curl -s http://localhost:5002/v2/restricted/mediawiki-multiversion/tags/list | jq '.tags' | wc -l
1752
elukey@registry1004:~$ curl -s http://localhost:5002/v2/restricted/mediawiki-webserver/tags/list | jq '.tags' | wc -l
527
elukey@registry1004:~$ curl -s http://localhost:5002/v2/restricted/mediawiki-multiversion-cli/tags/list | jq '.tags' | wc -l
1762
elukey@registry1004:~$ curl -s http://localhost:5002/v2/restricted/mediawiki-multiversion-debug/tags/list | jq '.tags' | wc -l
1764
elukey@registry1004:~$ curl -s http://localhost:5002/v2/restricted/mediawiki-singleversion/tags/list | jq '.tags' | wc -l
1481
elukey@registry1004:~$ curl -s http://localhost:5002/v2/restricted/mediawiki-singleversion-cli/tags/list | jq '.tags' | wc -l
1504
elukey@registry1004:~$ curl -s http://localhost:5002/v2/restricted/mediawiki-singleversion-debug/tags/list | jq '.tags' | wc -l
1505

We don't have SSDs yet on Ceph APUs so bucket listing etc.. is a little slower, plus the GC needs to do mark and sweep for all images/layers stored - the bigger the number of elements in the bucket the more time it takes IIUC.

It sounds like we can drop a lot of the above, so I think that GC timings may be reduced a lot :)

I tried to delete restricted images via docker-registryctl on build nodes, but the required credentials (ci-restricted) are not deployed yet. The build nodes have just the prod-build credentials, that allow to push/drop/etc.. all the other images but restricted. If we want to be able to drop restricted images as well, we'll have to either allow prod-build to drop those images too or to deploy ci-restricted on build hosts.

Change #1339621 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/docker-report@master] Allow registryctl to use a custom Docker config.json filepath

https://gerrit.wikimedia.org/r/1339621

In T437297#12308543, @elukey wrote:

I tried to delete restricted images via docker-registryctl on build nodes, but the required credentials (ci-restricted) are not deployed yet. The build nodes have just the prod-build credentials, that allow to push/drop/etc.. all the other images but restricted. If we want to be able to drop restricted images as well, we'll have to either allow prod-build to drop those images too or to deploy ci-restricted on build hosts.

The alternative could be to deploy docker-registryctl on deploy1003, and force it to use /etc/docker-pusher/config.json that contains the ci-restricted credentials. In this way we'd avoid to mix use cases, and Releng would have a way to periodically drop images with scap clean. The only missing feature is https://gerrit.wikimedia.org/r/c/operations/docker-images/docker-report/+/1339621 since docker-registryctl is not able to look up a custom config.json filepath yet.

Change #1339621 merged by Elukey:

[operations/docker-images/docker-report@master] Allow registryctl to use a custom Docker config.json filepath

https://gerrit.wikimedia.org/r/1339621

Change #1341832 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/docker-report@master] Release version 0.0.20

https://gerrit.wikimedia.org/r/1341832

Change #1341832 merged by Elukey:

[operations/docker-images/docker-report@master] Release version 0.0.20

https://gerrit.wikimedia.org/r/1341832

Created the new python3-docker-report deb and installed it on deploy1003 (manually for the moment), I was able to drop the following:

elukey@deploy1003:~$ sudo -i docker-registryctl -c /etc/docker-pusher/config.json  delete-tags docker-registry.discovery.wmnet/rest[4/4]
d/mediawiki-multiversion:2026-02*                                                                                                       
                                                                                                                                        
We're about to delete the following tags for image docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:                   
2026-02-26-140901-publish-83                                                                                                            
2026-02-26-142034-publish-83                                                                                                            
2026-02-26-143023-publish-83                                                                                                            
2026-02-26-144442-publish-83                                                                                                            
2026-02-26-145422-publish-83                                                                                                            
2026-02-26-153511-publish-83                                                                                                            
2026-02-26-164531-publish-83                                                                                                            
2026-02-26-165205-publish-83                                                                                                            
2026-02-26-172138-publish-83                                                                                                            
2026-02-26-190354-publish-83                                                                                                            
2026-02-26-210431-publish-83                                                                                                            
2026-02-26-211726-publish-83                                                                                                            
2026-02-26-213419-publish-83                                                                                                            
2026-02-26-214807-publish-83                                                                                                            
2026-02-26-215758-publish-83                                                                                                            
2026-02-26-220826-publish-83                                                                                                            
2026-02-27-142754-publish-83                                                                                                            
2026-02-27-153213-publish-83                                                                                                            
2026-02-27-225400-publish-83                                                                                                            
Ok to proceed? (y/n)y  
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-140901-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-142034-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-143023-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-144442-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-145422-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-153511-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-164531-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-165205-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-172138-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-190354-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-210431-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-211726-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-213419-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-214807-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-215758-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-26-220826-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-27-142754-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-27-153213-publish-83[DONE]
docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-02-27-225400-publish-83[DONE]

Caveat: docker-registryctl translates tags into sha256 and then issues DELETEs for the latter. Docker Distribution has implemented the drop in a very naive way: it lists all the sha256 manifests and then once the right is found, it drops it. And it does it for all tags separately, since we issue separate HTTP DELETE calls for every one of them. It may take a bit to drop data, and the larger the S3 bucket the slower the procedure is. I'll add more info later on when dropping other images, but let's keep this in mind when we'll think about adding it as part of (or right after) scap clean-images.

Change #1341899 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::k8s::deployment_server: add python3-docker-report

https://gerrit.wikimedia.org/r/1341899

elukey@deploy1003:~$ time sudo -i docker-registryctl -c /etc/docker-pusher/config.json  delete-tags docker-registry.discovery.wmnet/restricted/mediawiki-multiversion:2026-03*
[..]
real    358m55.156s
user    0m21.026s
sys     0m0.220s

That is a gigantic amount of time, but it can be easily explainable: we don't have SSDs for lookup on S3/apus, and docker-registryctl does a ton of S3 get content behind the scenes to find the right sha256 manifest to drop. I am not sure if the DELETE api supports bulk deletions, maybe it could be more efficient.

I dropped two more months and it took ~261m44.255s for the last block, so as expected removing stuff improves lookup times even when dropping.

Change #1341899 merged by Elukey:

[operations/puppet@production] profile::k8s::deployment_server: add python3-docker-report

https://gerrit.wikimedia.org/r/1341899

@dancy hi! I am close to complete the first cleanup of the old MediaWiki manifests, and now I'd like to try to run the GC workflow to drop real data. I'll try to provide more up-to-date timings but my gut feeling is that the first run may take a lot of hours to complete. My idea would be to scap lock, run GC and then finally release the lock, maybe over a weekend? Would it work if properly announced? Otherwise lemme know if you have other ideas. The main drawback of the scap lock over the weekend is what to do if we have an emergent deployment, because stopping the GC halfway through may not be ideal, but not sure if I am over-complicating it or not.

I ran the GC in dry-run mode and it completed in ~67 minutes, that is a nicer result than the first attempt. The time doesn't take into account the actual S3 delete api call:

elukey@registry1004:~$ grep "blob eligible for deletion" gc_run  | wc -l
33040

So maybe in total a couple of hours? Seems doable even in a MediaWiki maintenance window, but I'd run it on a day where deployments don't happen to avoid stalling too much deployers if things doesn't go as planned. The more we keep the bucket small the quicker the time will be, so at steady state we should probably take under an hour to run this.

Next steps:

• Run the first GC without dry-run to drop data up to the beginning of August.
• Start dropping manifests when scap cleans up old images.
• Second round of data drop (August plus part of September etc..) through a systemd timer or similar.

Change #1344634 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] docker_registry: rework read-only

https://gerrit.wikimedia.org/r/1344634

Mentioned in SAL (#wikimedia-operations) [2026-10-06T09:57:36Z] <elukey@deploy1003> Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297

Mentioned in SAL (#wikimedia-operations) [2026-10-06T09:57:55Z] <elukey@deploy1003> Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297 (duration: 00m 19s)

Mentioned in SAL (#wikimedia-operations) [2026-10-06T09:58:12Z] <elukey@deploy1003> Locking from deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297

Mentioned in SAL (#wikimedia-operations) [2026-10-06T10:02:56Z] <elukey> elukey@registry1004:~$ time sudo -u docker-registry /usr/bin/docker-registry garbage-collect /etc/docker/registry/config-restricted.yml -m | tee gc_run_06102026 - T437297

Mentioned in SAL (#wikimedia-operations) [2026-10-06T13:32:22Z] <elukey@deploy1003> Unlocked for deployment [MediaWiki]: Drop old MediaWiki images from the Registry - T437297 (duration: 214m 10s)

After ~130 mins the mark and sweep completed, and ~400GB of data has been dropped from S3 \o/

We should probably try to do the following:

• use registryctl or scap clean to drop images periodically
• run the GC periodically (maybe over weekends without holding the scap lock?) to keep the bucket stable/small-enough.

@dancy is the first item something that scap clean could do? Or maybe we could add a systemd timer on deploy1003 that does it separately, using docker-registryctl.