Needs discussion: do we like this approach? do the WMCS team have something else in mind?

Seems better than nothing. I'm actually pretty surprised to hear that something like this was not already in-place.

For the similar aptly system we document this manual rsync step:

$ sudo rsync --chmod 440 --chown root:"${INSTANCEPROJECT}".admin -ilrt /srv/packages/ /data/project/.system/aptly/"$(hostname -f)"

That target directory is on NFS so there is even a bit more redundancy in the DRBD cluster backing the /data/project share.

Not a reason to block this rsync idea at all, but one of the future use-cases I see for Ceph is providing durable shared storage for our internal Docker registry. See https://github.com/docker/docker-registry/blob/master/ADVANCED.md#non-amazon-s3-compliant-object-stores-eg-ceph-and-riak-cs

Yes, I already known about the aptly backup to the NFS. But honestly, I would try avoiding any solution that depends on NFS. +1 for Cephs.

Meanwhile, I think there is some puppet code I can refactor to introduce this rsync stuff "easily".

@aborrero this could definitely work but it will have IMO two issues:

• Between rsyncs invocations, registry could differ on content, so registry A had image B but it wasn't copied to registry B, depends of the particular image missed could lead to a substantial problem.
• While DNS is migrated there is no docker registry service available, if docker registry is behind a cache like Varnish, you can mitigate the effect of the main registry downtime caching some images. Keep in mind that unless you retag an image (something that shouldn't be done usually anyway) docker images are immutable so caching them will allow you to continue to serve images while you migrate the registry.

For maintaining images in sync you can consider to set up a docker registry hook that triggers some script to maintain the consistency between registries, if you want to follow this path you will need to set up notifications https://docs.docker.com/registry/notifications/ and code and API capable to interact with standby registry when it receives a new notification (you can use libraries like https://github.com/google/containerregistry or https://github.com/ivanilves/lstags ). Obviously, an rsync is a far simpler approach if you can afford the image propagation delay or missing out some image.

A shared network storage like Ceph, gluster or Openstack Swift (even NFS) will help you to achieve and active-active and horizontal scalable solution. https://github.com/minio/minio could also be used that offers an S3 compatible API.

You know better than me about the Toolforge environment and maybe the issues i described are not so critical there.

Change 483765 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: refactor docker registry profile

https://gerrit.wikimedia.org/r/483765

@fselles thanks for the advice. Comments inline

In T213695#4880789, @fselles wrote:

• Between rsyncs invocations, registry could differ on content, so registry A had image B but it wasn't copied to registry B, depends of the particular image missed could lead to a substantial problem.

We don't need/do load-balancing in the registry. We have a cold-standby/hot-spare model for this service. We have a machine ready to take over in case of disaster or maintenance. Other than that, a single machine can do all the work.

• While DNS is migrated there is no docker registry service available, if docker registry is behind a cache like Varnish, you can mitigate the effect of the main registry downtime caching some images. Keep in mind that unless you retag an image (something that shouldn't be done usually anyway) docker images are immutable so caching them will allow you to continue to serve images while you migrate the registry.

Also, registry updates are very low in frequency. They don't happen even weekly I would say. Therefore our use case is quite simple. DNS record updates (even counting on cache expiration) is enough for us I think.
We don't do load-balancing here, see above.

For maintaining images in sync you can consider to set up a docker registry hook that triggers some script to maintain the consistency between registries, if you want to follow this path you will need to set up notifications https://docs.docker.com/registry/notifications/ and code and API capable to interact with standby registry when it receives a new notification (you can use libraries like https://github.com/google/containerregistry or https://github.com/ivanilves/lstags ). Obviously, an rsync is a far simpler approach if you can afford the image propagation delay or missing out some image.

Yes, I think rsync is the way to go in this case. Provides a substantial improvement to the current situation with relative low time investment (compared to coding something new from scratch).

A shared network storage like Ceph, gluster or Openstack Swift (even NFS) will help you to achieve and active-active and horizontal scalable solution. https://github.com/minio/minio could also be used that offers an S3 compatible API.

I would really like to avoid NFS. We are trying to get rid of NFS in the mid term, so not adding more stuff depending on it is important. We are exploring Ceph. In fact, is one of our Q goals. But won't be on time for this. We need to improve the registry in the short term. Once we have Ceph in place, we can revisit the registry architecture.

Change 483765 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: refactor docker registry profile

https://gerrit.wikimedia.org/r/483765