Background
We currently run a docker registry on: https://docker-registry.wikimedia.org
See: https://wikitech.wikimedia.org/wiki/Docker-registry for more information about this.
This service forms a mission-critical piece of our production infrastructure and its use is ever increasing, as more workloads are moved to Kubernetes.
However, there are known issues with the current service.
Known issues
- T390251: docker-registry.wikimedia.org keeps serving bad blobs - This is a high-priority problem that is impacting various deployment pipeline, including mediawiki. The root cause is known and related to the current use of swift back-end, which is exhibiting replication latency between clusters, but it is difficult to fix as things stand.
- We currently have a limit of 4GB for layer sizes. (Ref: T404742#11194509 and T394778#10967688) - This limit is related to two components of the current system:
- The use of tmpfs on the nginx reverse proxies, which is acting as a blob cache.
- Swift imposes a 5 GB limit on the size of an individual object. Any uploads above that size must use multi-part uploads, but our current registry implementation does not do this.
- The use of a Swift backend is no longer supported by the latest version of the CNCF distribution registry software that we currently use. This means that we cannot upgrade without changing the back-end.
- Deleting old images is challenging and unreliable. See T375645 for details.
This slide deck goes into these issues in a little more detail: SRE & Security Summit 24-25 - Future steps for the Docker Registry
Existing solution proposals and ongoing work
There are several existing tickets that describe remedial work that we have discussed and planned, or are in the process of carrying out.
- T394476: Onboard the Docker Registry to apus - This involves switching the registry to a different back-end, based on the ceph/apus S3 services, operated by Data-Persistence
- T412951: Move the docker registry's /restricted prefix to Docker Distribution backed up by Ceph - This is a high-priority fix for the bad blobs problem, intended to mitigate the impact on the highest priority pipelines first by moving those to the ceph/apus S3 backend before all of the other images.
These solutions focus on keeping the docker-registry.wikimedia.org service running, whilst making iterative changes to address the known issues above.
This proposal
This proposal is somewhat different from the existing tickets, as it is intended to be worked on in parallel with them.
The suggested proposal is to design a new container registry service from scratch and build a PoC alongside the existing service.
This would allow us to address the known problems of the existing service, from the outset, but without the risk of negatively impacting operations.
We might reference this new system, for now, as: http://container-registry.wikimedia.org
It should be built with the aim of superseding the existing service, so we would expect all of the active image generation pipelines that publish to https://docker-registry.wikimedia.org to publish to this new service, eventually.
We should include within scope:
- Do we wish to continue to use the CNCF distribution, at version 3.0, or would another implementation be better?
- Alternative implementations include GitLab Container Registry, or Harbor, or Quay, or Sonatype Nexus, and likely more.
We could start by using this PoC service with the new ml-build1001 machine that has arisen from this requirement: T394778: Build and push images to the docker registry from ml-lab
We know from T385173 that the Machine-Learning-Team is currently able to keep image layers below 4GB and can therefore publish to the existing docker-registry.
However, it takes extra work to keep these layers small and the system is still somewhat brittle. If we were able to build this new container registry solution and onboard the ML team first
The work will require close collaboration across teams, since there are many stakeholders involved.
However, with the right communication channels in place, the work could continue at pace.
Acceptance Criteria
These acceptance criteria are ordered by a rough chronological order in which they are expected to be reached.
- A cross-team working group is formed to help design and implement the solution.
- A design for the PoC system is drawn up, reviewed, and ratified by the working group.
- The new container-registry service is running in the production realm and hardened appropriately.
- We have verified that deleting images works as designed.
- We have verified that performance of the new registry is satisfactory.
- Some container workloads from beta testers has been pushed and pulled, including layers larger than 5GB.
- A migration plan for the existing deployment pipelines is devised and, once again, ratified by the working group.
- The ML team is able to use this service for their vLLM or similar images, in production.
Failure criteria
- We failed to migrate workloads at least one image from each of the following classes within 9 months of the instantiation of the PoC
- The production-images pipeline.
- GitLab-CI trusted runners and the blubber/kokkuri pipelines.