Page MenuHomePhabricator

Auth fails for `docker-pusher` script on `contint1001`
Closed, ResolvedPublic

Description

From https://integration.wikimedia.org/ci/job/service-pipeline/32/console

[service-pipeline] Running shell script
+ sudo /usr/local/bin/docker-pusher docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid:build-32
The push refers to a repository [docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid]
6fa9c0522178: Preparing
0d111570ed17: Preparing
fae8af7d18e4: Preparing
0c03b4356347: Preparing
25ded9f9eec0: Preparing
90e7e7a8266b: Preparing
90e7e7a8266b: Waiting
error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>403 Forbidden</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx/1.11.13</center>\r\n</body>\r\n</html>\r\n"

Given that nginx, and not the docker daemon, is the one responding with a 403, perhaps our credentials are wrong or the user is not authorized. @hashar or maybe @akosiaris (someone with root on contint1001) can you verify that the contents of /etc/docker-pusher/config.json are populated with the right base64 encoded credentials for docker-registry.wikimedia.org?

Also, the job is attempting to push to a wikimedia/mediawiki-services-mathoid repo in the registry. Does this naming convention need tweaking at all and/or does the repo need to be initialized before we can push?

Event Timeline

dduvall triaged this task as High priority.Oct 19 2017, 6:09 PM
dduvall moved this task from Backlog to CI on the Release Pipeline board.

The reason why you're not currently able to upload to the registry is that it whitelists the clients that can upload images. I will need to add the contint machines, and maybe even create separate credentials for different namespaces. For now, i'll work with @hashar to fix this.

Change 385321 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] docker-registry: allow pushing images from other hosts

https://gerrit.wikimedia.org/r/385321

Change 385321 merged by Giuseppe Lavagetto:
[operations/puppet@production] docker-registry: allow pushing images from other hosts

https://gerrit.wikimedia.org/r/385321

I have rebuild it and it still 403. https://integration.wikimedia.org/ci/job/service-pipeline/33/console

Maybe the credentials do not let us push to wikimedia/ ?

Change 385342 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::docker::registry: allow using an external certificate

https://gerrit.wikimedia.org/r/385342

Change 385343 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/dns@master] Add entry for docker-registry.discovery.wmnet

https://gerrit.wikimedia.org/r/385343

Change 385343 merged by Giuseppe Lavagetto:
[operations/dns@master] Add entry for docker-registry.discovery.wmnet

https://gerrit.wikimedia.org/r/385343

I 've updated the credentials file as well, should now mention the new docker-registry.discovery.wmnet DNS. @hashar want to try again, with the new DNS and not the public read-only one ?

I have changed the registry in the Jenkins jobs: https://gerrit.wikimedia.org/r/#/c/380551/6..7/jjb/service-pipeline.groovy

Get https://docker-registry.discovery.wmnet/v2/:
x509: certificate is valid for darmstadtium.eqiad.wmnet, not docker-registry.discovery.wmnet

Change 385342 merged by Alexandros Kosiaris:
[operations/puppet@production] profile::docker::registry: allow using an external certificate

https://gerrit.wikimedia.org/r/385342

https://integration.wikimedia.org/ci/job/service-pipeline/35/console passed. We have to push to docker-registry.discovery.wmnet.

The push refers to a repository [docker-registry.discovery.wmnet/wikimedia/mediawiki-services-mathoid]
build-35: digest: sha256:6c151afe393291b9efdd28d6525bc7601a045b75bad451b630b80dbb1dc2a9c2 size: 9669

And I can retrieve it locally:

docker pull docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid@sha256:6c151afe393291b9efdd28d6525bc7601a045b75bad451b630b80dbb1dc2a9c2
sha256:6c151afe393291b9efdd28d6525bc7601a045b75bad451b630b80dbb1dc2a9c2: Pulling from wikimedia/mediawiki-services-mathoid
Status: Downloaded newer image for docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid@sha256:6c151afe393291b9efdd28d6525bc7601a045b75bad451b630b80dbb1dc2a9c2

Kudos to @Joe / @akosiaris