Page MenuHomePhabricator

docker-registry is returnning HTTP 403 Forbidden for all requests
Closed, ResolvedPublic

Description

https://integration.wikimedia.org/ci/job/composer-package-php72-docker/911/console

06:21:39 + exec docker run --rm --env-file /dev/fd/63 --volume /srv/jenkins-workspace/workspace/composer-package-php72-docker/log:/log --volume /srv/jenkins-workspace/workspace/composer-package-php72-docker/cache:/cache --volume /srv/jenkins-workspace/workspace/composer-package-php72-docker/src:/src docker-registry.wikimedia.org/releng/composer-package-php72:0.1.3
06:21:39 Unable to find image 'docker-registry.wikimedia.org/releng/composer-package-php72:0.1.3' locally
06:21:40 docker: Error response from daemon: error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>403 Forbidden</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx/1.13.6</center>\r\n</body>\r\n</html>\r\n".
06:21:40 See 'docker run --help'.
km@km-pt ~> curl "https://docker-registry.wikimedia.org/v2/_catalog"
<html>
<head><title>403 Forbidden</title></head>
<body bgcolor="white">
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.13.6</center>
</body>
</html>

Details

Related Gerrit Patches:

Event Timeline

Legoktm created this task.Aug 11 2018, 6:24 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 11 2018, 6:24 AM
Legoktm renamed this task from docker-registry returned HTTP 403 Forbidden in CI run to docker-registry is returnning HTTP 403 Forbidden for all requests.Aug 12 2018, 8:20 AM
Legoktm triaged this task as Unbreak Now! priority.
Legoktm updated the task description. (Show Details)
Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptAug 12 2018, 8:20 AM
Legoktm added a subscriber: Dzahn.Aug 12 2018, 8:23 AM

This is unbreak now from a CI perspective, I can't deploy or pull any new images.

The only recent puppet change I could find mentioning docker was rOPUPf7dcb27bb448: docker::registry: use ::profile::base::firewall by @Dzahn.

ema added a subscriber: ema.Aug 12 2018, 8:55 AM

This is due to the move of cache_misc sites to cache_text T164609.

There seems to be some type of ACL on darmstadtium.eqiad.wmnet for docker-registry checking where the HTTP request comes from:

From cp1045, part of the cache_misc cluster:

08:54:38 ema@cp1045.eqiad.wmnet:~
$ curl -v "Host: docker-registry.wikimedia.org" http://darmstadtium.eqiad.wmnet:81/ 2>&1 | grep HTTP
> GET / HTTP/1.1
< HTTP/1.1 404 Not Found

From cp1079, member of cache_text:

08:55:29 ema@cp1079.eqiad.wmnet:~
$ curl -v "Host: docker-registry.wikimedia.org" http://darmstadtium.eqiad.wmnet:81/ 2>&1 | grep HTTP
> GET / HTTP/1.1
< HTTP/1.1 403 Forbidden

Change 452182 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] profile::docker::registry: whitelist cache_text nodes

https://gerrit.wikimedia.org/r/452182

Change 452182 merged by Ema:
[operations/puppet@production] profile::docker::registry: whitelist cache_text nodes

https://gerrit.wikimedia.org/r/452182

ema closed this task as Resolved.Aug 12 2018, 9:16 AM
ema claimed this task.

@Legoktm confirmed that the issue is now solved, closing.