We have moved special Docker Registry's prefixes to separate S3 backends:
In the latter we just started from scratch pushing new Docker images to it, meanwhile in the former we used a script to copy over Docker images from Swift to the new S3 backend. The parent task's problems (bad blobs etc..) have not resurfaced yet, and we have leveraged a nice and more flexible replication between eqiad and codfw.
This task is about the creation of new S3 backends:
- releng - move all the images prefixed with /releng
- main - move all the other images that are not included in /releng or /ml or /restricted
In theory we could be more granular and create more buckets, but:
- We are currently holding ~6TB of Docker images on Swift, but we are probably using 20% of them. Docker Distribution has never implemented a true workflow to do deep storage clean, and its garbage collection workflow didn't work on Swift (it doesn't do deep cleanup as we wish, but it does *something*). Our initial concern with apus/S3 was that listing buckets with hundreds of objects could lead to slow downs, but it shouldn't be the case imho. Last but not least, medium to long term we are planning to add SSDs to apus, so I don't think it will be a problem even for later use cases.
- As Janis mentioned in T420978#11945134, we need to be mindful about how we split the images into buckets because we may loose re-using common layers across images. For example, we cannot have one or more base images or common layers hosted on Swift and the final layers on S3, the Docker client wouldn't be able to pull the layers.
I would personally start migrating over to main only the Docker images running on K8s clusters, that are easily retrievable via Debmonitor, we may have few other use cases in production but I don't think more than that (excluding the releng prefix of course).