Current docker registry instance is writing to a Swift container in codfw, to achieve HA we would like to maintain a "global" docker registry container synched in each DC. Giving the nature of what we are writing (docker layers) some delay over the synchronization will be acceptable.
The main idea to implement this is to use container sync feature and create two new containers (docker_registry_eqiad and docker_registry_codfw) configured to be mutually synchronized, so a write on the codfw cluster will be replicated over eqiad and the other way around.
There is a bunch of open questions related to this task:
- How container synchronization will perform, there are signs that it was tested and discarded before.
- How monitor container synchronization, naive answer will be log synchronization logging to another file and configure log forwarder to ship to ELK, also a simple list diff check in icinga. It seems there are no easy ways to monitor it as stated in the doc.
Additionally, it should be noted there is no way for an end user to detect sync progress or problems other than HEADing both containers and comparing the overall information
- How to manage containers creation in Puppet, it seems that current swift containers are created by applications and only swift accounts and config are manage via Puppet, not containers.