Page MenuHomePhabricator

Move the majority of the Registry's docker image prefixes to a new s3 bucket
Closed, ResolvedPublic

Description

After moving releng's, restricted's and ml's prefixes to separate buckets, it is the turn of the Docker images that run on Kubernetes (and few misc prod deployments).

My idea is to do the following:

  • For every k8s cluster, use the following one-liner to get the unique image:tag combinations of all running containers.
kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec['initContainers', 'containers'][*].image}" |tr -s '[[:space:]]' '\n' |sort |uniq
  • Make a unique list across all clusters, and use the registry-clone script to copy over the aforementioned image:tag combinations plus all the tags that, when ordered using Debian versioning etc.., come later (up to latest for example).
  • Copy base and production-images over as well, using a cut-off that is meaningful (namely, by default only the 2026 image tags except older ones that may be mentioned in the various repos, like puppet and deployment-charts).
  • Verify if puppet and deployment-charts list images that need to be copied.

Once we have the list, we can copy over and then flip the nginx config during a special maintenance window. It will be easy to copy over missing images via skopeo (see https://wikitech.wikimedia.org/wiki/Docker-registry#Storage_backends) so anybody on-call will be able to quickly fix issues.

Event Timeline

Change #1327549 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::docker_registry: create the "main" docker distribution instance

https://gerrit.wikimedia.org/r/1327549

Change #1327549 merged by Elukey:

[operations/puppet@production] profile::docker_registry: create the "main" docker distribution instance

https://gerrit.wikimedia.org/r/1327549

elukey triaged this task as High priority.
LSobanski changed the task status from Open to In Progress.Aug 24 2026, 2:13 PM
LSobanski moved this task from Incoming to Work in Progress on the Infrastructure-Foundations board.

After a lot of filtering for non useful logs I found something good-enough:

elukey@registry2004:~$ sudo journalctl --since '1 days ago' -u docker-registry-swift.service | egrep -v "(GET|HEAD) /v2/(restricted/.*|dev/.*|wikimedia/machinelearning.*|releng.*|repos/releng.*) HTTP/1.1"| grep -v 'HTTP/1.1" 404' | egrep -o '((GET|HEAD) /v2/.* HTTP/1.1)' | grep -v 'HTTP/1.1\" 404' | grep "/manifests/" | sed -e "s/\/manifests\//:/g" | sed -e "s/sha256:/sha256@/g" | cut -d " " -f 2 | cut -d "?" -f 1 | sort | uniq
/v2/abstractwiki-rust:latest
/v2/bookworm:20260510
/v2/bookworm:20260621
/v2/bookworm:20260802
/v2/bookworm:20260816
/v2/bookworm:latest
/v2/bookworm:sha256@5477057f01713e2aa77c97930ee32e409286dceed2730073ced6908baf370ad9
/v2/bookworm:sha256@95ad5ae01417d0ee743a898533be406899da0db2a1e74a71cba9a5cb1b8a4955
/v2/bookworm:sha256@ce64c83b25df5db984340d5e83effbf36090d8db58984fd64d050e6123172394
/v2/bullseye:latest
/v2/bullseye:sha256@82718e4adda9bcb38fb45a12585351add0197dee4eee787baa74b89d8ea90c92
/v2/buster:latest
/v2/calico/kube-controllers:v3.29.1
/v2/calico/node:v3.29.1
/v2/calico/node:v3.30.7
/v2/calico/typha:v3.29.1
/v2/calico/typha:v3.30.7
/v2/ceph:18.2.8-wmf1
/v2/ceph:sha256@f568f535317e027d2cae54b4ed5c5edbef1f310e444326fbec0d71b7a3841db2
/v2/cfssl-issuer:0.3.0-1
/v2/cfssl-issuer:0.3.1-0
/v2/cfssl-issuer:0.4.0-1-20260104
/v2/cfssl-issuer:sha256@01445853b88c449b9ad013c503703984aa3122ccbeab4b917d50fcd740c4ed4d
/v2/coredns1.11:1.11.3-2
/v2/coredns1.11:sha256@ae62991caa89a711f6a4bd702e7b0196e31bd03020f2d3825249c9e463ad69ee
/v2/coredns1.12:1.12.1-1
/v2/coredns:sha256@9b4d1e420d4ca8864764a26142e36c2b4ae6096e48d51d233fd984329ab6408f
/v2/envoy:1.35.9-1
/v2/envoy:1.39.0-1
/v2/envoy:latest
/v2/envoy:sha256@a9f7d5cb943ad89c60aea9701ee5427b9f71bbdab20b04c46c554269c8538f17
/v2/eventrouter:0.3.0-6
/v2/eventrouter:0.4.0-1
/v2/eventrouter:0.4.0-2
/v2/eventrouter:0.4.0-3-20260816
/v2/eventrouter:sha256@5ceddb6b347c795785d4ce5d27c77c738a97ecec86be2e2d8f2c33520d61e88b
/v2/eventrouter:sha256@d6c7bb7bd4b668c57152c61af7833343098be5a9d9e784145b030ea6fe82f840
/v2/eventrouter:sha256@ef3f05ee9f6aa4f5394874361f954521b72f2ccec945a9564774e1f8d2ed0a04
/v2/flink:1.20.3-wmf1-20260510
/v2/flink:2.0.2-wmf1
/v2/flink:sha256@b7e45884ef9df11cab41fd8ed9ff4ef39439e896b26cf3c60bd4e7dfcad6838d
/v2/flink:sha256@fd679593b450a2f55f20c85deb2c6c70fcce03adec55d340e69178b90af6b65b
/v2/golang1.24:1.24-20260823
/v2/golang1.24:sha256@7f5b008bf382fed81f5387b9f36fd23e1744b5559614cbd02f07a5c8825c38a5
/v2/golang1.25:latest
/v2/haproxy:3.2.15-1
/v2/haproxy:latest
/v2/haproxy:sha256@13a9a3249b2dbc65f401f1bbb615c9164b23cf87b5459cdf25118fcf0c8c8a07
/v2/httpd-fcgi:2.4.67-1
/v2/httpd-fcgi:latest
/v2/httpd-fcgi:sha256@b63b457e123162d217bb3b7a0d4cfb6ef99b5439a1f8b4d0d14ecfdf40a76da4
/v2/knative-serving-activator:1.21.2-1
/v2/knative-serving-controller:1.21.2-1
/v2/knative-serving-webhook:1.21.2-1
/v2/kserve-controller:0.11.2-2
/v2/kube-state-metrics2.18:2.18.0-1-20260802
/v2/mcrouter:2023.07.17.00-2-20260726
/v2/mcrouter:sha256@2727186fb4f80cca79188b4248ab2aaadc529d10ac69734dfbe13958b2a11e83
/v2/mediawiki-httpd:latest
/v2/memcached:latest
/v2/nodejs18-devel:latest
/v2/nodejs18-devel:sha256@b496966003d785a905a688c4d9631c29c760c0a31b9f2e2afbba85b2df01acad
/v2/nodejs20-slim:latest
/v2/nodejs20-slim:sha256@255319155a7218403e749a5eb1c60224e09cd487933cf00f6fe6ad4f2110a076
/v2/nodejs22-slim:latest
/v2/nodejs22-slim:sha256@1263c12896fc07ae5029f40291a4570f96b14be39db16a89cef4036e49b52b0b
/v2/nodejs24-slim:latest
/v2/nodejs24-slim:sha256@8b6116fff765ea90b87fa19914101aab37968ebc5a65da13ec83f59713a1d824
/v2/nutcracker:0.0.5-20240825
/v2/nutcracker:sha256@c04013ec4263462c8c2c347a8b86971cd428ea82508698c0860b9bf4b4762653
/v2/openjdk-8-jdk-bookworm:8.462-1-20260621
/v2/pause:3.6-1
/v2/pause:sha256@4274f278b6895d4dab4b7d486a4933a59656a0a968214f7129f870f20a7a07db
/v2/php8.3-fpm-multiversion-base:latest
/v2/php8.3-fpm-multiversion-base:sha256@c09ce1364542fa4e735280ec78466f276b13f2016d04cd63c44b127f5acb57cb
/v2/php8.5-fpm-multiversion-base:latest
/v2/prometheus-apache-exporter:0.0.4
/v2/prometheus-apache-exporter:sha256@2ab6b197b145a28d677032ba9e5df5e851231a88acb3b7cdddc7071166a02b3a
/v2/prometheus-apache-exporter:sha256@72c65ce6e28dfc855f9dc9f779c63e15179d2f1d0613c8bacbd83481d2318d01
/v2/prometheus-mcrouter-exporter:sha256@fb4d67e10b1bf2d4091f5696e611d93367053e75ef6aa3fd02aa21509b3f812b
/v2/prometheus-php-fpm-exporter:0.0.4
/v2/prometheus-php-fpm-exporter:0.0.4-20240623
/v2/prometheus-php-fpm-exporter:sha256@01a04e92eafd78afa444d1b76fe06a62d24e8d5ebfc70742314d47527babd5c9
/v2/prometheus-php-fpm-exporter:sha256@2d40ad10ed75fb4e3e7154070c001222700a099d6633a91ae5198005ee65ac64
/v2/prometheus-statsd-exporter:0.26.1-2-20250907
/v2/prometheus-statsd-exporter:latest
/v2/prometheus-statsd-exporter:sha256@be5b4c230af9e573792dbb4f4b61cc727d73318ee312430a20707cbfdbd3b2bb
/v2/python3-bookworm:latest
/v2/python3-bookworm:sha256@0dc54ce4a9e4fb2861ed1c717c6e59c4921082ad7175ac868152449ccedea12e
/v2/python3-build-bookworm:latest
/v2/python3-build-bookworm:sha256@4542674007dca1edb163ca7b374247193046b0a56ee7f54180c0cfeab5d7b654
/v2/python3-build-trixie:latest
/v2/python3-build-trixie:sha256@2422202b8fc051f2585abd4bfffb16eae24337969af9981c510c0968f77f6699
/v2/python3-devel:0.0.4-20250727-s1-20260614
/v2/python3-trixie:latest
/v2/python3-trixie:sha256@db65c260806d2d72df75160be709cbd549417189d99a240be6615e01c86a477e
/v2/ratelimit:9.0.4-20260107.e9ce92c
/v2/ratelimit:sha256@33932ff12208ec1c28bab397aab1cca717f0bbd8aeebc2d639c761b6c033faad
/v2/repos/abstract-wiki/ci-images/curl-jq:latest
/v2/repos/abstract-wiki/ci-images/curl-jq:sha256@7bca693017f47f6933b5e92b99be801d5942c136e776f32ef76e520f5570d496
/v2/repos/abstract-wiki/ci-images/helm:latest
/v2/repos/abstract-wiki/wikifunctions/function-evaluator/rusty-js:latest
/v2/repos/abstract-wiki/wikifunctions/function-evaluator/rusty-js:sha256@826d554173731e898fb3ede807d891d34162517eb5b0556e9968a06e11a160b6
/v2/repos/abstract-wiki/wikifunctions/function-evaluator/rusty-py:latest
/v2/repos/abstract-wiki/wikifunctions/function-evaluator/rusty-py:sha256@4774562b3e020dceed9ac1195a787b7eb97447128fdf75e48ed7faf15a935b9c
/v2/repos/abstract-wiki/wikifunctions/function-evaluator/wasm-javascript-all:latest
/v2/repos/abstract-wiki/wikifunctions/function-evaluator/wasm-javascript-all:sha256@4dfb88548689d222fdce50b53911ddd5b1306add2471527e2eb378c676a28001
/v2/repos/abstract-wiki/wikifunctions/function-evaluator/wasm-python3-all:latest
/v2/repos/abstract-wiki/wikifunctions/function-evaluator/wasm-python3-all:sha256@9ece0e9784381bcc38bfb288a9146b1a7c077fa992ec2294c01831faf7e5d816
/v2/repos/abstract-wiki/wikifunctions/function-orchestrator:latest
/v2/repos/abstract-wiki/wikifunctions/function-orchestrator:sha256@138b464ae48a7451ec6d860e52b64eaf4f6827b708567f19d5fb6d07e4b4c0d7
/v2/repos/cloud/cloud-vps/horizon/deploy:2025-09-19-161533
/v2/repos/cloud/cloud-vps/horizon/deploy:2026-07-07-214712
/v2/repos/data-engineering/airflow-dags:latest
/v2/repos/data-engineering/airflow-dags:sha256@18e5bf4bbd07aa0a8eb9a93d7f1dc52640f16d7bc973b3cca6398ea2c868702d
/v2/repos/data-engineering/airflow:latest
/v2/repos/data-engineering/blunderbuss:sha256@1094847900f362504d7f75ccc4e1864e7062784ddc8170999745f19a80fa3e6e
/v2/repos/data-engineering/blunderbuss:sha256@7a1cf422b34b6ccde4b47161bb3ed4c2f22900975f099a9ac748a10ad46fa323
/v2/repos/data-engineering/datahub/frontend:latest
/v2/repos/data-engineering/datahub/gms:latest
/v2/repos/data-engineering/datahub/mae-consumer:latest
/v2/repos/data-engineering/datahub/mce-consumer:latest
/v2/repos/data-engineering/datahub/upgrade:latest
/v2/repos/data-engineering/eventgate-wikimedia:configurable-transforms-fc7a6596
/v2/repos/data-engineering/eventgate-wikimedia:v1.26.0
/v2/repos/data-engineering/eventgate-wikimedia:v1.30.0
/v2/repos/data-engineering/eventstreams:node24-update-693cfcc5
/v2/repos/data-engineering/ferretdb:latest
/v2/repos/data-engineering/git-sync:sha256@3e01121704b405a08649012571aba0ce6834ab3aa3428df0b02a476b7ba4c3f5
/v2/repos/data-engineering/growthbook:5.0.0-2026-08-18-132846-335f36322e3d805267c29f2c5a9e8b00b1868851
/v2/repos/data-engineering/growthbook:latest
/v2/repos/data-engineering/kerberos-kinit:45377f59c5bdf8bae1b967c49ee29a144c5cba44-production
/v2/repos/data-engineering/kerberos-kinit:latest
/v2/repos/data-engineering/kerberos-kinit:sha256@3d8b08daaaa1bf7444fdf613c14e860e64b927088f49ff93ff004a1da1835ab3
/v2/repos/data-engineering/kerberos-kinit:sha256@5a48a0fa4d9fe6ac95023b36da50a10677c8ca1857994d69e878263574702a2a
/v2/repos/data-engineering/opensearch:2
/v2/repos/data-engineering/opensearch:2025-09-02-211140-2ef1c90115e12fc0a475a1b15c358c3657a5348b
/v2/repos/data-engineering/opensearch:2026-05-13-143357-2b2e022d3e2756b0dbd31d66e341a587f5d3b8d7-production2
/v2/repos/data-engineering/opensearch-operator:2025-08-25-204922-014d2b0b6a65947de4486c3919c5b5dcce45d39e
/v2/repos/data-engineering/opensearch-operator:sha256@41631f251b4617e4d2b7b68b52499439527580ae5551c387fc73964a341f992b
/v2/repos/data-engineering/spark/spark3.4-history:sha256@79be7ee5df18824f8c63a7f9bd8efbf00528f8aabe3431592db3af827107ea53
/v2/repos/data-engineering/superset/superset-backend:sha256@6f3ecf31783259d64f9dcdad2402dd0577a84e0ab574f31b97d02a40d5eb02c3
/v2/repos/data-engineering/superset/superset-frontend:sha256@8f2e622d1be62a1c494cc7a53d88955a1f0a5b66e191a433a15410e53e0f8fa3
/v2/repos/data-engineering/test-kitchen:latest
/v2/repos/mediawiki/services/chart-renderer:latest
/v2/repos/mediawiki/services/editcheck-headless:latest
/v2/repos/mediawiki/services/ipoid:latest
/v2/repos/mediawiki/services/mathoid:latest
/v2/repos/mediawiki/services/zotero:latest
/v2/repos/search-platform/cirrussearch-elasticsearch-image:sha256@e3e1896c36ce732b71c727106c4682fb99bb37f20d969e8dbbf44ec474e91b8e
/v2/repos/search-platform/cirrussearch-elasticsearch-image:v7.10.2-11
/v2/repos/search-platform/cirrussearch-elasticsearch-image:v7.10.2-12
/v2/repos/search-platform/cirrussearch-elasticsearch-image:v7.10.2-5
/v2/repos/search-platform/cirrussearch-opensearch-image:sha256@b4de34bcb80208df33a6ebed678c3df8ade1029b06cd7bb198a47589ec8c284d
/v2/repos/search-platform/cirrussearch-opensearch-image:v1.3.20-12
/v2/repos/search-platform/cirrussearch-opensearch-image:v2.19.5-3
/v2/repos/test-platform/catalyst/catalyst-ci-client:latest
/v2/repos/test-platform/catalyst/catalyst-ci-client:sha256@23c7e9efeda4badc2a680e0d99aa82ed1d63146e39bf068b3ae26ea07695e212
/v2/repos/test-platform/catalyst/patchdemo:2e1283c7
/v2/repos/test-platform/catalyst/patchdemo:e3ce00bb
/v2/repos/test-platform/catalyst/patchdemo:latest
/v2/repos/wikidata-platform/wdqs/wdqs-backend-health-monitor:latest
/v2/repos/wikidata-platform/wdqs/wdqs-proxy:latest
/v2/repos/wikidata-platform/wdqs/wdqs-qlever:latest
/v2/repos/wikidata-platform/wikidata-platform-tools:latest
/v2/rsyslog:8.2504.0-1
/v2/rsyslog:sha256@9e8d9ca45f943f86d83cadbba84af80b8daf63982907361d1b72b2432daafb86
/v2/trixie:20260607
/v2/trixie:20260823
/v2/trixie:latest
/v2/trixie:sha256@41d74f80c5bc15a012e6058e0b99f1c1212201be1c41c2a71b6d0e48014ebf18
/v2/trixie:sha256@797d667a90378583f37c793bc76ea617c997d302c9acf16a93a67149d044abd7
/v2/wikimedia/labs-striker:2026-08-15-164102-production
/v2/wikimedia/mediawiki-libs-shellbox:2024-06-13-133425-video
/v2/wikimedia/mediawiki-libs-shellbox:2026-08-20-124436-media83
/v2/wikimedia/mediawiki-libs-shellbox:2026-08-20-124436-php-rpc83
/v2/wikimedia/mediawiki-libs-shellbox:2026-08-20-124436-score83
/v2/wikimedia/mediawiki-libs-shellbox:2026-08-20-124436-syntaxhighlight83
/v2/wikimedia/mediawiki-libs-shellbox:2026-08-20-124436-timeline83
/v2/wikimedia/mediawiki-libs-shellbox:2026-08-20-124436-video83
/v2/wikimedia/mediawiki-libs-shellbox:score83
/v2/wikimedia/mediawiki-libs-shellbox:sha256@0516702bd57a0a55fa172882e1dda55c5141949f3e077cb82130fc626b0e3f6f
/v2/wikimedia/mediawiki-libs-shellbox:sha256@79b84781dd7cf19bebfa4a99323361a5be4e2c1e5504931eeb079f61426b8ac6
/v2/wikimedia/mediawiki-libs-shellbox:sha256@85ee416f8af38f76fe006041c0a6997af9dbd4b326ac20599066481e93a13e8f
/v2/wikimedia/mediawiki-libs-shellbox:sha256@8f8937d09480e5ea34e2a1fab7d8d1bec573fb2df1cde0ca6e4cd005f0fdf846
/v2/wikimedia/mediawiki-libs-shellbox:sha256@9c49fb690cc13462c385ca40c4301ef072c2a64da58e4676cc0ae257f91e650a
/v2/wikimedia/mediawiki-libs-shellbox:sha256@e02dd1ff8f7a4f7e55c2262f9376f7fad45e551fbd6488fb108c84f04b19b1c7
/v2/wikimedia/mediawiki-services-change-propagation:2025-03-06-075118-production
/v2/wikimedia/mediawiki-services-chromium-render:2024-02-05-181957-production
/v2/wikimedia/mediawiki-services-chromium-render:2026-08-25-100601-production
/v2/wikimedia/mediawiki-services-chromium-render:sha256@2cf09b45b87a99ba59b3a4c90701fdd154dca3eeb1d61f5d3d0e9a478e234d73
/v2/wikimedia/mediawiki-services-chromium-render:sha256@4b94ebe86a2a583a50a84bbd9a8222435b418b152bc4430dde99a565c5c23d90
/v2/wikimedia/mediawiki-services-citoid:2019-04-01-104952-production
/v2/wikimedia/mediawiki-services-citoid:sha256@f4b790c8edc6a2f36febc7a1ba7ba125432db9a1236b53781242da457f0d9365
/v2/wikimedia/mediawiki-services-cxserver:2019-05-08-064536-production
/v2/wikimedia/mediawiki-services-cxserver:2026-08-24-032908-production
/v2/wikimedia/mediawiki-services-cxserver:sha256@0d094dddb2fd0137afe9c54764d53ae277b325f742d48c7db9677b0d1c3ac46a
/v2/wikimedia/mediawiki-services-kask:v1.0.10
/v2/wikimedia/mediawiki-services-mobileapps:2025-09-10-110204-production
/v2/wikimedia/mediawiki-services-push-notifications:2021-08-20-154251-publish
/v2/wikimedia/mediawiki-services-wikifeeds:2023-11-28-215911-production
/v2/wikimedia/mediawiki-services-zotero:2019-02-01-074657-candidate
/v2/wikimedia/mediawiki-services-zotero:sha256@6337cfe23e85a6a4aeb9f9b5b55e1eb17463343d614da23afad1915b62a12f93
/v2/wikimedia/operations-software-tegola:2026-05-05-083231-publish
/v2/wikimedia/operations-software-tegola:sha256@c58ed0a79376d82c957624bedab78d2b3dbc156145363a22058567e2725b7dcf
/v2/wikimedia/operations-software-thumbor-plugins:2023-10-06-153556-production
/v2/wikimedia/operations-software-thumbor-plugins:2026-08-19-234524-production
/v2/wikimedia/operations-software-thumbor-plugins:sha256@f987e3e1a472f0b12a0587c8631cdf41becaf02c67f179141e85c40269b79781
/v2/wikimedia/research-mwaddlink:2026-07-17-002610-production
/v2/wikimedia/research-mwaddlink:sha256@729ac938bd5dfd2ac1c2f504142c52a00f40edd33e8cd1250db42cf3b508d536
/v2/wikimedia/research-mwaddlink:stable
/v2/wikimedia/wikibase-termbox:latest
/v2/wikimedia/wikimedia-developer-portal:latest
/v2/wikimedia/wikimedia-toolhub:2026-07-21-105254-production
/v2/wikimedia/wikimedia-toolhub:latest
/v2/wikimedia/wikimedia-toolhub:sha256@6129b64760fafbc81eccd332fb2561e644d654d12b2e65a525e2409825e54267
/v2/wmf-debci-bookworm:latest

From a quick look it seems that 99% of the images are kubernetes-deploy/related, except little things like /v2/repos/cloud/cloud-vps/horizon/deploy that probably don't need a separate S3 bucket.

My plan is to use two sources to find what to copy:

  • Every image deployed on all Kubernetes clusters.
  • The available logs on the Docker Registry nodes.

I uploaded the images running on all k8s clusters (with the "future" tags up to latest) and I ended up with the following split:

elukey@registry1004:~$ s3cmd -c s3-apus-docker.cfg du -H
 302G   56063 objects s3://registry-main/
 263G    7112 objects s3://registry-ml/
  94G   25173 objects s3://registry-releng/
 633G  115758 objects s3://registry-restricted/

We are still missing some images (need to go through the same process for the Docker Registry's logs) but I don't anticipate a lot more GBs to be used. This is already a big win from the actual ~6TBs of swift usage, so we know now that the footprint that we need to run our clusters is way less.

It is interesting to notice that the registry-restricted bucket is already around 633GBs. We migrated the MediaWiki images to S3/registry-restricted-bucket months ago, so their footprint is definitely growing fast. Long term we need to come up with a good clean up strategy, otherwise we'll end up in the 6TB-swift-situation again during the next years.

To keep archives happy - I am using the following two bash scripts on deploy1003 to get the list of docker image names and tags (in sha256@etc.. form) for all k8s clusters:

#!/bin/bash

for cluster in eqiad codfw dse-k8s-eqiad aux-k8s-eqiad aux-k8s-codfw ml-serve-eqiad ml-serve-codfw ml-staging-codfw staging
do
    sudo ./dump_docker_images_k8s.sh $cluster
done
#!/bin/bash

set -e

source /etc/profile.d/kube-env.sh

if [ -z "$1" ]; then
    echo "Please pass the k8s cluster as arg."
fi

kube_env admin $1
kubectl get pods -A -o jsonpath="{.items[*].status['initContainerStatuses', 'containerStatuses'][*].imageID}" | tr -s '[[:space:]]' '\n' | sort | uniq

Then I use https://gitlab.wikimedia.org/elukey/registry-clone to map those (|sort|uniq etc..) to a set of image:tag combinations to use with Skopeo to copy images.

Very interestingly, the registry clone script proposes to copy 2019 image tags for things like citoid and cxserver. After a bit of log review I found out that very old deployment-prep VMs (on bullseye, so about to be nuked) still configured to pull old images. So now the question is if we want to copy old tags just to support that use case or not.

Change #1333237 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] docker_registry: move away from Swift completely

https://gerrit.wikimedia.org/r/1333237

Change #1333237 merged by Elukey:

[operations/puppet@production] docker_registry: move away from Swift completely

https://gerrit.wikimedia.org/r/1333237

Mentioned in SAL (#wikimedia-operations) [2026-09-07T12:14:36Z] <elukey> moved most of the Docker Registry's prefixes to a new internal S3 backend. For any docker pull failure that worked in the past, please ping me or drop a note in T435499 or contact the oncall SREs

Change #1337614 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::docker_registry: remove swift when generating image lists

https://gerrit.wikimedia.org/r/1337614

Change #1337614 merged by Elukey:

[operations/puppet@production] profile::docker_registry: remove swift when generating image lists

https://gerrit.wikimedia.org/r/1337614

Nothing to report after the switch, the metrics look good on the Registry's dashboard and nobody came up with issues so far (some MW deployments went through etc..). I am going to check for more traces of problems in the Registry's logs to be sure :)

I'll leave this task open for some days since there are multiple levels of caches to consider (for example, the k8s' Dragonfly one) and issues may surface later on.

Going to call this done since I didn't see any issue so far. Please re-open in case you spot something!

Hi @elukey

today I was missing some images from the docker-registry related to zuul.

I setup a new machine and it started with releng/zuul/zuul/zuul-executor can't be found.

When looking closer it looks like there is no releng/zuul/zuul structure at all anymore.

So the affected ones would be:

repos/releng/zuul/zuul/zuul-scheduler
repos/releng/zuul/zuul/zuul-executor
repos/releng/zuul/zuul/zuul-launcher
repos/releng/zuul/zuul/zuul-web

I asked about it and was reminded of your email about this.

Is it possible these were affected by the move? Thanks!

Hey @Dzahn! It should be fixed now!

For the record this is what I've done (from https://wikitech.wikimedia.org/wiki/Docker-registry#Storage_backends):

# tmux sesson on registry1004
sudo -u docker-registry /usr/bin/docker-registry serve /home/elukey/config-swift.yml
# notice the dest port 5006, because this is the Releng bucket
skopeo copy --all --preserve-digests docker://localhost:5000/repos/releng/zuul/zuul/zuul-scheduler:wmf-14.2.0-1 docker://localhost:5006/repos/releng/zuul/zuul/zuul-scheduler:wmf-14.2.0-1 --dest-tls-verify=false --src-tls-verify=false
skopeo copy --all --preserve-digests docker://localhost:5000/repos/releng/zuul/zuul/zuul-launcher:wmf-14.2.0-1 docker://localhost:5006/repos/releng/zuul/zuul/zuul-launcher:wmf-14.2.0-1 --dest-tls-verify=false --src-tls-verify=false
skopeo copy --all --preserve-digests docker://localhost:5000/repos/releng/zuul/zuul/zuul-executor:wmf-14.2.0-1 docker://localhost:5006/repos/releng/zuul/zuul/zuul-executor:wmf-14.2.0-1 --dest-tls-verify=false --src-tls-verify=false
skopeo copy --all --preserve-digests docker://localhost:5000/repos/releng/zuul/zuul/zuul-web:wmf-14.2.0-1 docker://localhost:5006/repos/releng/zuul/zuul/zuul-web:wmf-14.2.0-1 --dest-tls-verify=false --src-tls-verify=false

I copied only the last tag, wmf-14.2.0-1, because it was the only one listed in puppet afaics. Please let me know if I need to copy more! Sorry for the trouble.