Page MenuHomePhabricator

Toolforge: introduce some HA mechanism for the docker registry
Closed, ResolvedPublic

Description

The current situation in the Toolforge docker registry is that if a disaster happens (VM shutdown, data corruption, etc), we will have to rebuild and push all docker images to the registry, which may take a lot of time (downtime).

The proposed new HA method for the Toolforge docker registry is simple, cold-standby:

  • docker images are built and they are pushed to the active registry node (using the DNS docker-registry.tools.wmflabs.org)
  • the active registry node stores the image locally (usually /srv/registry)
  • there is a daily cron job running in the standby node to rsync the registry data from the active node.
  • in case of disaster of the active registry node (VM shutdown, corruption, etc), we can switch the main DNS and start the docker registry daemon in the standby node
  • we may loss the differential data in the registry since the last sync. That can be solved easily by pushing again the docker images, but only a few instead of all of them.

This cold-standby mechanism, even if not perfect from the automation point of view, provides a robust improvement with regards the current situation.
Also, is really simple to implement. The missing bits for this are:

Details

Related Gerrit Patches:
operations/puppet : productiontoolforge: refactor docker registry profile

Event Timeline

aborrero renamed this task from Toolforge: introduce some HA for the docker registry to Toolforge: introduce some HA mechanism for the docker registry.Jan 14 2019, 12:32 PM
aborrero triaged this task as High priority.
aborrero created this task.
aborrero updated the task description. (Show Details)
aborrero updated the task description. (Show Details)Jan 14 2019, 12:35 PM
aborrero updated the task description. (Show Details)

Needs discussion: do we like this approach? do the WMCS team have something else in mind?

Seems better than nothing. I'm actually pretty surprised to hear that something like this was not already in-place.

For the similar aptly system we document this manual rsync step:

$ sudo rsync --chmod 440 --chown root:"${INSTANCEPROJECT}".admin -ilrt /srv/packages/ /data/project/.system/aptly/"$(hostname -f)"

That target directory is on NFS so there is even a bit more redundancy in the DRBD cluster backing the /data/project share.

Not a reason to block this rsync idea at all, but one of the future use-cases I see for Ceph is providing durable shared storage for our internal Docker registry. See https://github.com/docker/docker-registry/blob/master/ADVANCED.md#non-amazon-s3-compliant-object-stores-eg-ceph-and-riak-cs

Yes, I already known about the aptly backup to the NFS. But honestly, I would try avoiding any solution that depends on NFS. +1 for Cephs.

Meanwhile, I think there is some puppet code I can refactor to introduce this rsync stuff "easily".

fsero added a subscriber: fsero.Jan 15 2019, 1:13 PM

@aborrero this could definitely work but it will have IMO two issues:

  • Between rsyncs invocations, registry could differ on content, so registry A had image B but it wasn't copied to registry B, depends of the particular image missed could lead to a substantial problem.
  • While DNS is migrated there is no docker registry service available, if docker registry is behind a cache like Varnish, you can mitigate the effect of the main registry downtime caching some images. Keep in mind that unless you retag an image (something that shouldn't be done usually anyway) docker images are immutable so caching them will allow you to continue to serve images while you migrate the registry.

For maintaining images in sync you can consider to set up a docker registry hook that triggers some script to maintain the consistency between registries, if you want to follow this path you will need to set up notifications https://docs.docker.com/registry/notifications/ and code and API capable to interact with standby registry when it receives a new notification (you can use libraries like https://github.com/google/containerregistry or https://github.com/ivanilves/lstags ). Obviously, an rsync is a far simpler approach if you can afford the image propagation delay or missing out some image.

A shared network storage like Ceph, gluster or Openstack Swift (even NFS) will help you to achieve and active-active and horizontal scalable solution. https://github.com/minio/minio could also be used that offers an S3 compatible API.

You know better than me about the Toolforge environment and maybe the issues i described are not so critical there.

Change 483765 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: refactor docker registry profile

https://gerrit.wikimedia.org/r/483765

aborrero added a comment.EditedJan 16 2019, 12:13 PM

@fselles thanks for the advice. Comments inline

In T213695#4880789, @fselles wrote:
  • Between rsyncs invocations, registry could differ on content, so registry A had image B but it wasn't copied to registry B, depends of the particular image missed could lead to a substantial problem.

We don't need/do load-balancing in the registry. We have a cold-standby/hot-spare model for this service. We have a machine ready to take over in case of disaster or maintenance. Other than that, a single machine can do all the work.

  • While DNS is migrated there is no docker registry service available, if docker registry is behind a cache like Varnish, you can mitigate the effect of the main registry downtime caching some images. Keep in mind that unless you retag an image (something that shouldn't be done usually anyway) docker images are immutable so caching them will allow you to continue to serve images while you migrate the registry.

Also, registry updates are very low in frequency. They don't happen even weekly I would say. Therefore our use case is quite simple. DNS record updates (even counting on cache expiration) is enough for us I think.
We don't do load-balancing here, see above.

For maintaining images in sync you can consider to set up a docker registry hook that triggers some script to maintain the consistency between registries, if you want to follow this path you will need to set up notifications https://docs.docker.com/registry/notifications/ and code and API capable to interact with standby registry when it receives a new notification (you can use libraries like https://github.com/google/containerregistry or https://github.com/ivanilves/lstags ). Obviously, an rsync is a far simpler approach if you can afford the image propagation delay or missing out some image.

Yes, I think rsync is the way to go in this case. Provides a substantial improvement to the current situation with relative low time investment (compared to coding something new from scratch).

A shared network storage like Ceph, gluster or Openstack Swift (even NFS) will help you to achieve and active-active and horizontal scalable solution. https://github.com/minio/minio could also be used that offers an S3 compatible API.

I would really like to avoid NFS. We are trying to get rid of NFS in the mid term, so not adding more stuff depending on it is important. We are exploring Ceph. In fact, is one of our Q goals. But won't be on time for this. We need to improve the registry in the short term. Once we have Ceph in place, we can revisit the registry architecture.

aborrero closed this task as Resolved.Jan 16 2019, 4:40 PM

The rsync service is working fine.

Change 483765 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: refactor docker registry profile

https://gerrit.wikimedia.org/r/483765

aborrero updated the task description. (Show Details)Jan 16 2019, 5:53 PM