Page MenuHomePhabricator

Design and build the next generation of container-registry service for the WMF production realm
Open, MediumPublic

Description

Background

We currently run a docker registry on: https://docker-registry.wikimedia.org
See: https://wikitech.wikimedia.org/wiki/Docker-registry for more information about this.

This service forms a mission-critical piece of our production infrastructure and its use is ever increasing, as more workloads are moved to Kubernetes.
However, there are known issues with the current service.

Known issues
  1. T390251: docker-registry.wikimedia.org keeps serving bad blobs - This is a high-priority problem that is impacting various deployment pipeline, including mediawiki. The root cause is known and related to the current use of swift back-end, which is exhibiting replication latency between clusters, but it is difficult to fix as things stand.
  2. We currently have a limit of 4GB for layer sizes. (Ref: T404742#11194509 and T394778#10967688) - This limit is related to two components of the current system:
    1. The use of tmpfs on the nginx reverse proxies, which is acting as a blob cache.
    2. Swift imposes a 5 GB limit on the size of an individual object. Any uploads above that size must use multi-part uploads, but our current registry implementation does not do this.
  3. The use of a Swift backend is no longer supported by the latest version of the CNCF distribution registry software that we currently use. This means that we cannot upgrade without changing the back-end.
  4. Deleting old images is challenging and unreliable. See T375645 for details.

This slide deck goes into these issues in a little more detail: SRE & Security Summit 24-25 - Future steps for the Docker Registry

Existing solution proposals and ongoing work

There are several existing tickets that describe remedial work that we have discussed and planned, or are in the process of carrying out.

  1. T394476: Onboard the Docker Registry to apus - This involves switching the registry to a different back-end, based on the ceph/apus S3 services, operated by Data-Persistence
  2. T412951: Move the docker registry's /restricted prefix to Docker Distribution backed up by Ceph - This is a high-priority fix for the bad blobs problem, intended to mitigate the impact on the highest priority pipelines first by moving those to the ceph/apus S3 backend before all of the other images.

These solutions focus on keeping the docker-registry.wikimedia.org service running, whilst making iterative changes to address the known issues above.


This proposal

This proposal is somewhat different from the existing tickets, as it is intended to be worked on in parallel with them.

The suggested proposal is to design a new container registry service from scratch and build a PoC alongside the existing service.
This would allow us to address the known problems of the existing service, from the outset, but without the risk of negatively impacting operations.

We might reference this new system, for now, as: http://container-registry.wikimedia.org

It should be built with the aim of superseding the existing service, so we would expect all of the active image generation pipelines that publish to https://docker-registry.wikimedia.org to publish to this new service, eventually.

We should include within scope:

We could start by using this PoC service with the new ml-build1001 machine that has arisen from this requirement: T394778: Build and push images to the docker registry from ml-lab

We know from T385173 that the Machine-Learning-Team is currently able to keep image layers below 4GB and can therefore publish to the existing docker-registry.
However, it takes extra work to keep these layers small and the system is still somewhat brittle. If we were able to build this new container registry solution and onboard the ML team first

The work will require close collaboration across teams, since there are many stakeholders involved.
However, with the right communication channels in place, the work could continue at pace.

Acceptance Criteria

These acceptance criteria are ordered by a rough chronological order in which they are expected to be reached.

  • A cross-team working group is formed to help design and implement the solution.
  • A design for the PoC system is drawn up, reviewed, and ratified by the working group.
  • The new container-registry service is running in the production realm and hardened appropriately.
  • We have verified that deleting images works as designed.
  • We have verified that performance of the new registry is satisfactory.
  • Some container workloads from beta testers has been pushed and pulled, including layers larger than 5GB.
  • A migration plan for the existing deployment pipelines is devised and, once again, ratified by the working group.
  • The ML team is able to use this service for their vLLM or similar images, in production.

Failure criteria

  • We failed to migrate workloads at least one image from each of the following classes within 9 months of the instantiation of the PoC
    • The production-images pipeline.
    • GitLab-CI trusted runners and the blubber/kokkuri pipelines.

Event Timeline

Thanks for this Ben.

Overall, this is similar to what I had wanted to include in this FY's Annual Planning. I had also wanted to define the audience, well supported uses and metrics. Unfortunately, it did not make the cut and now we face the consequences.

This is light on technical details, which is acceptable, leaving the working group to discover and decide on the best technical approach. So, overall 👍 , with the following caveat:

We could start by using this PoC service with the new ml-build1001 machine that has arisen from this requirement: T394778: Build and push images to the docker registry from ml-lab

This increases the risk of a Second System Effect. Namely:

  1. New system comes up, designed radically different from old system to avoid old system's problems.
  2. New system serves a need the old system can't serve, proves useful
  3. Migration starts to the new system from the old system.
  4. Old system serves needs that new system can't serve (because of unforeseen complications, too much work, too many stakeholders, too large scope, you name it)
  5. We are stuck with both systems, higher maintenance costs, higher friction/confusion and so on.

I would suggest to explicitly de-risk this by avoiding New system serves a need the old system can't serve, proves useful from the list. Migrate something easy, then something really hard (e.g. MediaWiki), migrate more stuff and THEN prove really useful by serving a need the old can't serve.
Similarly, I would move the A migration plan for the existing deployment pipelines is devised and, once again, ratified by the working group. above the vLLM acceptance criteria (I am assuming here there is some implicit order, if not, let's be explicit there is no order. Or alternatively let's be explicit what the order is). Given T385173: Use rocm/vllm image on Lift Wing it looks like it won't be a deal breaker to do so.

And I would also add a set of failure criteria, e.g. we failed to migrate workloads X, Y and Z by <insert date> (+ some extension/leeway if applicable), we abandon the new registry.

Note: If the above sounds eerily reminiscent of Gitlab vs Gerrit, IPv4 vs IPv6, or RESTBase vs MediaWiki, it's because they are examples of the 2nd system effect.

There is also an opportunity here to try to consolidate the technology used for container registries around the foundation, for example there is also the Toolforge one. If in the long term we end up using the same stack for both should simplify their operations and maintenance and allow to have more people familiar with the stack.

There is also an opportunity here to try to consolidate the technology used for container registries around the foundation, for example there is also the Toolforge one. If in the long term we end up using the same stack for both should simplify their operations and maintenance and allow to have more people familiar with the stack.

Yes, I 've been thinking of that angle too. Toolforge hosts 2 different registries. At least 1 of those, if not both, make some sense to be collocated with the main one. This is one of the things I allude to above with I had also wanted to define the audience, well supported uses and metrics

I would suggest to explicitly de-risk this by avoiding New system serves a need the old system can't serve, proves useful from the list. Migrate something easy, then something really hard (e.g. MediaWiki), migrate more stuff and THEN prove really useful by serving a need the old can't serve.

Yes, that seems like a good plan.
I was thinking that the large layers issue was particularly pressing, so the current vLLM requirement would be a good early use-case. but I take your point that migrating workload in the order of:

  • easy -> difficult -> currently unsupported -> remaining

...is likely a better recipe for success.

Similarly, I would move the A migration plan for the existing deployment pipelines is devised and, once again, ratified by the working group. above the vLLM acceptance criteria (I am assuming here there is some implicit order, if not, let's be explicit there is no order. Or alternatively let's be explicit what the order is).

Done. I added a note about implicit but not strict chronological ordering, but we can always revisit this.

And I would also add a set of failure criteria, e.g. we failed to migrate workloads X, Y and Z by <insert date> (+ some extension/leeway if applicable), we abandon the new registry.

Done. Again, maybe not the final version, but a start.

LSobanski triaged this task as Medium priority.Feb 9 2026, 3:48 PM

[who is expect{ing,ed} to own the storage backend for this service?]

[who is expect{ing,ed} to own the storage backend for this service?]

@MatthewVernon It will very likely be anything that exposes an S3 interface/API, so apus is probably the first candidate that will be proposed. Let me know your thoughts!

To keep archives happy - docker distribution seems to work really nicely with S3/apus and all the previous issues with Swift's eventual consistency didn't re-occur (at least for now).

Now it comes something really important to decide - shall we start the evaluation work now, and possibly move Docker images to say harbor via tools like this, or should we move away from swift to s3 on docker distrubution and then do the comparison work?

I am very torn since moving away from Swift would be a huge win, and the work to be done to migrate away within docker distribution itself may not be that much (we are using 6TB of data on Swift and 80% of them are garbage from the past).

For example, we could do the following:

  • create s3-docker-registry.wikimedia.org - a new endpoint that points to a new docker distribution instance where s3/apus are the first storage configured (no swift)
  • we scan the image:tag combinations on our Wikikube clusters and CI, and then:
# /v2/batman is used on WIkikube or in production
docker pull docker-registry.wikimedia.org/v2/batman
docker tag docker-registry.wikimedia.org/v2/batman s3-docker-registry.wikimedia.org/v2/batman
docker push s3-docker-registry.wikimedia.org/v2/batman

At the end we flip s3-docker-registry.wikimedia.org to docker-registry.wikimedia.org. It would transparently move the storage behind the scenes, leaving aside all the garbage that we have on Swift that we don't use. It would also be a good exercise if we have to do it again in the future.

I'm currently trying to get some quotes to put an expansion request in for next FY for apus, primarily to enable us to have a small solid-state-only storage pool to use for bucket indexes and similar, which should give us better performance and reliability with buckets with larger numbers of objects in.

Since the last update a lot of things changed, going to make a summary here :)

  • The Docker Registry is now being serve entirely from an internal S3 API, backed by the apus Ceph cluster.
  • We now have 4 different docker distribution endpoints on each registry VM, with nginx in front routing requests. The idea is the following:
    • Requests to /v2/wikimedia/machine-learning and /v2/repos/machinelearning are routed to a dedicated ML bucket/instance. There is an nginx config to set the 5GB layer limit, client_max_body_size 4608m; that can be overridden at the location block level, so in the future if ML needs it we'll be able to upgrade only this part of the Docker Registry's URI paths (of course it'll need a bump in the os' tmpfs partition first).
    • Requests to /v2/releng and /v2/repos/releng will go to a dedicated Releng bucket.
    • Requests to /v2/restricted (MediaWiki private images) will go to a separate bucket.
    • All the rest will go to a "main" catch-all bucket, that in theory should represent most of what deployed on Kubernetes.
  • The footprint of the image binaries are reduced to ~2.5TB (compared to the ~6TB used on Swift) because we didn't copy everything in the copy process (Swift -> S3 buckets).
  • Skopeo is an amazing tool to copy Docker images between Docker distribution endpoints without extra temporary storage and risk of altering the image's sha256 manifest in the process.
  • The Docker Distribution's GC workflow seems working without early errors (as it happened with Swift), so we now may thing about dropping old binaries for real.

What are the remaining challenges? Is it still worth to investigate a new container registry solution?

The major remaining challenge is about dropping old image binaries from S3. When we do https://wikitech.wikimedia.org/wiki/Docker-registry#Deleting_images, only the manifests are dropped from S3, meanwhile the layer binaries etc.. are kept. The docker distribution binary can also run as garbage collector, doing a mark&sweep pass over all the files on S3 and dropping what's not referenced/linked anymore. Caveat - the GC workflow needs to be run when the Registry is in read-only mode, because new images being pushed may end up being incorrectly flagged by mark&sweep and deleted (or partially deleted).

I started T437297 to evaluate a full GC run on the restricted bucket, since it is easier to avoid images being pushed (you should run scap lock and block deployments). The task should inform us about how the GC behaves, but having a regular cleanup of old stuff would really be nice. For example, harbor uses Docker Distribution behind the scenes but also Postgres, and it seems way better in handling this.

Last caveats to keep in mind when evaluating a new solution:

  • We need to find a tool able to offer a read-only view for the outside Internet, but not for our internal network. And also a way to control what we publish to the outside word as available images (the docker registry's UI).
  • We use https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/tree/main via Nginx to authenticate clients that need to build gitlab repos, since we need JWT for those (not simply Basic Auth).

My personal take - we are in a much better position with the Registry backed up by S3, so we are not in any immediate rush to move to something different. The apus cluster should be fine in allowing a couple more Terabites of space used, that we'll fill in a long time, so at this current rate of growth we shouldn't be pressured to upgrade for this reason either. But long term having a way to specify a yaml config about what to drop and when, without requiring us to manually execute cookbooks/scripts/spells/etc.. would really be nice :)

My personal take - we are not in a much better position with the Registry backed up by S3, so we are not in any immediate rush to move to something different.

Did you mean "in a much better position" rather than "not in..."?

My personal take - we are not in a much better position with the Registry backed up by S3, so we are not in any immediate rush to move to something different.

Did you mean "in a much better position" rather than "not in..."?

Yep fixed sorry!

Updates from T437297 - I was able to drop ~400GB of deleted Docker image blobs/layers/etc.. from the registry bucket, while holding the scap lock as precaution to avoid new images to be pushed. It took ~2 hours, and I think the time heavily depends on how big the S3 bucket is. So the GC on S3 code seems to work fine!