Page MenuHomePhabricator

Auth fails for `docker-pusher` script on `contint1001`
Closed, ResolvedPublic

Description

From https://integration.wikimedia.org/ci/job/service-pipeline/32/console

[service-pipeline] Running shell script
+ sudo /usr/local/bin/docker-pusher docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid:build-32
The push refers to a repository [docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid]
6fa9c0522178: Preparing
0d111570ed17: Preparing
fae8af7d18e4: Preparing
0c03b4356347: Preparing
25ded9f9eec0: Preparing
90e7e7a8266b: Preparing
90e7e7a8266b: Waiting
error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>403 Forbidden</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx/1.11.13</center>\r\n</body>\r\n</html>\r\n"

Given that nginx, and not the docker daemon, is the one responding with a 403, perhaps our credentials are wrong or the user is not authorized. @hashar or maybe @akosiaris (someone with root on contint1001) can you verify that the contents of /etc/docker-pusher/config.json are populated with the right base64 encoded credentials for docker-registry.wikimedia.org?

Also, the job is attempting to push to a wikimedia/mediawiki-services-mathoid repo in the registry. Does this naming convention need tweaking at all and/or does the repo need to be initialized before we can push?

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 19 2017, 6:09 PM
dduvall triaged this task as High priority.Oct 19 2017, 6:09 PM
dduvall moved this task from Backlog to CI on the Release Pipeline board.
Joe added a comment.Oct 20 2017, 6:09 AM

The reason why you're not currently able to upload to the registry is that it whitelists the clients that can upload images. I will need to add the contint machines, and maybe even create separate credentials for different namespaces. For now, i'll work with @hashar to fix this.

Joe claimed this task.Oct 20 2017, 6:27 AM

Change 385321 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] docker-registry: allow pushing images from other hosts

https://gerrit.wikimedia.org/r/385321

Change 385321 merged by Giuseppe Lavagetto:
[operations/puppet@production] docker-registry: allow pushing images from other hosts

https://gerrit.wikimedia.org/r/385321

I have rebuild it and it still 403. https://integration.wikimedia.org/ci/job/service-pipeline/33/console

Maybe the credentials do not let us push to wikimedia/ ?

Change 385342 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] profile::docker::registry: allow using an external certificate

https://gerrit.wikimedia.org/r/385342

Change 385343 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/dns@master] Add entry for docker-registry.discovery.wmnet

https://gerrit.wikimedia.org/r/385343

Change 385343 merged by Giuseppe Lavagetto:
[operations/dns@master] Add entry for docker-registry.discovery.wmnet

https://gerrit.wikimedia.org/r/385343

I 've updated the credentials file as well, should now mention the new docker-registry.discovery.wmnet DNS. @hashar want to try again, with the new DNS and not the public read-only one ?

I have changed the registry in the Jenkins jobs: https://gerrit.wikimedia.org/r/#/c/380551/6..7/jjb/service-pipeline.groovy

Get https://docker-registry.discovery.wmnet/v2/:
x509: certificate is valid for darmstadtium.eqiad.wmnet, not docker-registry.discovery.wmnet

Change 385342 merged by Alexandros Kosiaris:
[operations/puppet@production] profile::docker::registry: allow using an external certificate

https://gerrit.wikimedia.org/r/385342

hashar closed this task as Resolved.Oct 20 2017, 10:51 AM

https://integration.wikimedia.org/ci/job/service-pipeline/35/console passed. We have to push to docker-registry.discovery.wmnet.

The push refers to a repository [docker-registry.discovery.wmnet/wikimedia/mediawiki-services-mathoid]
build-35: digest: sha256:6c151afe393291b9efdd28d6525bc7601a045b75bad451b630b80dbb1dc2a9c2 size: 9669

And I can retrieve it locally:

docker pull docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid@sha256:6c151afe393291b9efdd28d6525bc7601a045b75bad451b630b80dbb1dc2a9c2
sha256:6c151afe393291b9efdd28d6525bc7601a045b75bad451b630b80dbb1dc2a9c2: Pulling from wikimedia/mediawiki-services-mathoid
Status: Downloaded newer image for docker-registry.wikimedia.org/wikimedia/mediawiki-services-mathoid@sha256:6c151afe393291b9efdd28d6525bc7601a045b75bad451b630b80dbb1dc2a9c2

Kudos to @Joe / @akosiaris