Page MenuHomePhabricator

Move the Docker Registry's /ml prefix to S3/apus
Closed, ResolvedPublic

Description

We have been running with the /v1/restricted prefix handled by a separate docker registry instance, and I'd like to do the same with the /v1/ml one since we have a dedicated Docker registry instance only for it. Main goals:

  1. Separate ML's requirements from the rest, so that we'll be able in the future to bump their maximum layer size limit if needed (client_max_body_size can be specified in Nginx's location contextes, so we could ideally bump the tmpfs without changing the overall restrictions).
  2. Test a procedure to migrate existing images to S3/apus.

I had an idea for the second point, that is not super nice and clean but it seems simple and effective:

  1. We pull the /ml/vllm image on a build host.
  2. We tag the image in a way that it can be pushed to registry2005.codfw.wmnet:$PORT (with $PORT set with the ML Registry's listening port)
  3. We allow buildXXXX to push to registry2005 via that port, using an iptables rule.
  4. We push the image, that will end up on S3/apus.
  5. We flip the nginx config to use the new backend.

Last but not the least, we'll need to update the Python script that generates the home dir for docker-registry.wikimedia.org, to include the new source (otherwise it won't get published and it will be confusing).

Event Timeline

elukey added a subscriber: Scott_French.

Change #1286961 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::docker_registry: rename ml bucket

https://gerrit.wikimedia.org/r/1286961

Change #1286961 merged by Elukey:

[operations/puppet@production] profile::docker_registry: rename ml bucket

https://gerrit.wikimedia.org/r/1286961

I moved all the https://docker-registry.wikimedia.org/ml images (vllm, various versions) to apus using the above trick. Now it is a matter of doing the following:

  1. update the docker registry's UI code to pull image names from more backends, at the moment it defaults to the switft one.
  2. remove the vllm images from the docker registry, so they will not show up anymore on the UI.
  3. flip nginx config to use the new ml backend on registry nodes.

Change #1287292 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] docker_registry: allow multiple docker instances

https://gerrit.wikimedia.org/r/1287292

I had a chat with @JMeybohm the other day, and he pointed out a very wise thing - when we switch one image to its new S3-backend, we'll likely duplicate the amount of time a Docker base image is stored as layer. Let's make a concrete example with ML: the vLLM images are using amd-pythorch-common as base image, that in turns uses python3-bookworm. These base images are currently stored (namely, docker-pull-able) in the Swift backend because nginx dictates the routes, so once I pushed the vLLM images to the ML S3 backend, those layers had to be pushed as well. They won't figure as available repositories (in the Docker Distribution sense) in the ML backend, but just another layer. This means that a k8s worker (the Docker client) will potentially have to pull the same base image layer multiple times, if running ML images and other images running from the main Swift backend. This is a trade off that we have already been using/accepting for the /restricted prefix, so it is not the end of the world, but it should be kept in mind when figuring out the overall final number of S3 backends to use (for all the images, not only the ML ones, so another task's concern).

Change #1287292 merged by Elukey:

[operations/puppet@production] docker_registry: allow multiple docker instances

https://gerrit.wikimedia.org/r/1287292

Change #1290808 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] docker_registry: move the /ml prefix to its new S3 backend

https://gerrit.wikimedia.org/r/1290808

Change #1290808 merged by Elukey:

[operations/puppet@production] docker_registry: move the /ml prefix to its new S3 backend

https://gerrit.wikimedia.org/r/1290808

elukey claimed this task.

Change deployed! I tested a Docker pull and everything looks good. I think this is a very nice first step towards the goal of moving everything to APUs. I am going to open another task to track the work of moving the other images from Swift to S3.

elukey@build2001:~$ docker pull docker-registry.wikimedia.org/ml/amd-vllm014:gfx90agfx942rocm7.0.0pytorch2.10.0mori0.1flash-attn2.8.3aiter0.1.7vllm0.14
gfx90agfx942rocm7.0.0pytorch2.10.0mori0.1flash-attn2.8.3aiter0.1.7vllm0.14: Pulling from ml/amd-vllm014
13547a5c281d: Pull complete 
a25cdfaaf59e: Pull complete 
31eed947e2d7: Pull complete 
6f20a16d8fd8: Pull complete 
ec5c862ee382: Pull complete 
62c4059a163d: Pull complete 
d60118ba7375: Pull complete 
82e181487223: Pull complete 
8a26556285bb: Pull complete 
006f0dff2581: Pull complete 
9fd31a2f6f6a: Pull complete 
c3de556ba0bc: Pull complete 
af47d837c39d: Pull complete 
Digest: sha256:96b84b5e8ba6b6088df36ba4f99e27d551b396c236d5f761b2f041f62fbf0ad4
Status: Downloaded newer image for docker-registry.wikimedia.org/ml/amd-vllm014:gfx90agfx942rocm7.0.0pytorch2.10.0mori0.1flash-attn2.8.3aiter0.1.7vllm0.14
docker-registry.wikimedia.org/ml/amd-vllm014:gfx90agfx942rocm7.0.0pytorch2.10.0mori0.1flash-attn2.8.3aiter0.1.7vllm0.14

Change #1295371 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] docker_registry: remove duplicates from registry-homepage-builder.py

https://gerrit.wikimedia.org/r/1295371

Change #1295371 merged by Elukey:

[operations/puppet@production] docker_registry: remove duplicates from registry-homepage-builder.py

https://gerrit.wikimedia.org/r/1295371