On contint1001, using docker-pkg, I have created a new image docker-registry.wikimedia.org/releng/tox:0.4.0.
On WMCS instance, I am unable to pull that image, the manifest is not found.
Eventually after sometime it appeared on the catalog at: https://tools.wmflabs.org/dockerregistry/releng/tox/tags/
Later on attempt to fetch yields things such as:
0.4.0: Pulling from releng/tox 8d22d214682d: Already exists dd5d82f356b7: Already exists 6adceec4ad0e: Pulling fs layer 95587f1fdaf7: Pulling fs layer dadbf101c83c: Pulling fs layer 6adceec4ad0e: Retrying in 5 seconds error pulling image configuration: unknown blob
0.4.0: Pulling from releng/tox 8d22d214682d: Already exists dd5d82f356b7: Already exists 6adceec4ad0e: Pulling fs layer 95587f1fdaf7: Pulling fs layer dadbf101c83c: Pulling fs layer error pulling image configuration: received unexpected HTTP status: 503 Backend fetch failed
On contint1001, I can pull the image if I refer to docker-registry.discovery.wmnet.
@fsero explained it might be due to a slow cross dc replication.
I see something slightly different when I try to pull locally:
docker pull docker-registry.wikimedia.org/releng/quibble-stretch-php70:0.0.31-3
Error response from daemon: manifest for docker-registry.wikimedia.org/releng/quibble-stretch-php70:0.0.31-3 not found: manifest unknown: manifest unknown
This is ~60 minutes after the image was built.