Error pulling image from docker registry
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Clarakosi
	Aug 27 2019, 9:30 PM

Description

When attempting to pull Kask v1.0.3 image from docker-registry.wikimedia.org I get the following error:

Error response from daemon: received unexpected HTTP status: 502 connect failed

The error was traced to pulling the manifest from the registry; e.g., curl https://docker-registry.wikimedia.org/v2/wikimedia/mediawiki-services-kask/manifests/v1.0.3

Snippet of the error html:

<p>See the error message at the bottom of this page for more&nbsp;information.</p>
</div>
</div>
<div class="footer"><p>If you report this error to the Wikimedia System Administrators, please include the details below.</p><p class='text-muted'><code>Request from 2601:406:4300:872d:c042:21e9:5c83:e275 via cp1075.eqiad.wmnet, ATS/8.0.5<br>Error: 502, connect failed at 2019-08-27 21:22:22 GMT</code></p></div>

Details

	Subject	Repo	Branch	Lines +/-
	Revert "ATS: temporarily use plain HTTP to access docker-registry"	operations/puppet	production	+1 -2
	ATS: temporarily use plain HTTP to access docker-registry	operations/puppet	production	+2 -1

Customize query in gerrit

Related Objects

Mentioned In: T231423: cergen fails signing CSR
Mentioned Here: T231423: cergen fails signing CSR

Event Timeline

• Clarakosi created this task.Aug 27 2019, 9:30 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 27 2019, 9:30 PM

Tagging in Traffic; this is the server (cp1075) running ATS not Varnish, right?

Assigning to @ema to investigate (yes, this is the live test server for ATS backends for these servers). Most likely the problem is specific to ATS<->docker-registry, probably because the underlying service TLS certificate's SAN list doesn't match the public name docker-registry.wikimedia.org.

This is blocking CI runs.

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptAug 27 2019, 9:44 PM

ayounsi subscribed.Aug 27 2019, 9:47 PM

This comment was removed by ayounsi.

Depooled cp1075 ats-be service via confctl, can someone retry and confirm mitigated?

In T231388#5443941, @BBlack wrote:

Depooled cp1075 ats-be service via confctl, can someone retry and confirm mitigated?

It works!

Please leave this open for now so @ema can look at a more-permanent fixup tomorrow!

De-prioritising.

• ema mentioned this in T231423: cergen fails signing CSR.Aug 28 2019, 9:33 AM

A proper fix for this issue is blocked on cergen bug T231423.

I am going to disable TLS between ATS and eqiad's docker-registry for the time being. cp1075 is also in eqiad, so there's no cross-DC traffic that can be snooped on anyways.

Change 532953 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: temporarily use plain HTTP to access docker-registry

https://gerrit.wikimedia.org/r/532953

gerritbot added a project: Patch-For-Review.Aug 28 2019, 9:42 AM

Change 532953 merged by Ema:
[operations/puppet@production] ATS: temporarily use plain HTTP to access docker-registry

https://gerrit.wikimedia.org/r/532953

Maintenance_bot removed a project: Patch-For-Review.Aug 28 2019, 2:10 PM

Change 533041 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] Revert "ATS: temporarily use plain HTTP to access docker-registry"

https://gerrit.wikimedia.org/r/533041

gerritbot added a project: Patch-For-Review.Aug 28 2019, 3:24 PM

Change 533041 merged by Ema:
[operations/puppet@production] Revert "ATS: temporarily use plain HTTP to access docker-registry"

https://gerrit.wikimedia.org/r/533041

We have managed to generate a proper certificate for the docker-registry origin servers, and cp1075 is now back to using TLS to connect to them.

$ curl -v --resolve docker-registry.wikimedia.org:443:208.80.154.224 https://docker-registry.wikimedia.org/v2/wikimedia/mediawiki-services-kask/manifests/v1.0.3
[...]
> GET /v2/wikimedia/mediawiki-services-kask/manifests/v1.0.3 HTTP/2
> Host: docker-registry.wikimedia.org
[...]
< HTTP/2 200 
< date: Thu, 29 Aug 2019 10:45:21 GMT
< content-type: application/vnd.docker.distribution.manifest.v1+prettyjws
[...]
< x-cache: cp1075 miss, cp1077 miss
[...]
{
   "schemaVersion": 1,
   "name": "wikimedia/mediawiki-services-kask",
   "tag": "v1.0.3",
   "architecture": "amd64",

Maintenance_bot removed a project: Patch-For-Review.Aug 29 2019, 11:10 AM

Error pulling image from docker registryClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

Error pulling image from docker registry
Closed, ResolvedPublic
Actions