Page MenuHomePhabricator

Fix nginx config and caching for docker registry
Open, LowPublic

Description

Some responses from our docker-registry get cached in CDN, others don't.

  • Requests for the catalog are not cached
    • curl -I -XGET 'https://docker-registry.wikimedia.org/v2/_catalog
  • Requests for tag lists are cached
    • curl -I -XGET 'https://docker-registry.wikimedia.org/v2/envoy-tls-local-proxy/tags/list
  • Requests for manifests are cached (but the cache does not seem to honor the Accept header send by the client T242200)
    • curl -i -XGET -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' 'https://docker-registry.wikimedia.org/v2/envoy-tls-local-proxy/manifests/dontuseme'
    • curl -i -XGET 'https://docker-registry.wikimedia.org/v2/envoy-tls-local-proxy/manifests/sha256:0786b049723e8e9877e0939798dd8f5aaf6a2cd01b194f92d2d5a349e186c335'
  • HTTP 404 seem to get cached as well
    • curl -i -XGET -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' 'https://docker-registry.wikimedia.org/v2/envoy-tls-local-proxy/manifests/foobar'
    • curl -i -XGET 'https://docker-registry.wikimedia.org/v2/envoy-tls-local-proxy/manifests/sha256:0786b049723e8e9877e0939798dd8f5aaf6a2cd01b194f92d2dccccccccccccc'
  • Requests for blobs are cached (404's here as well)
    • curl -I -XGET 'https://docker-registry.wikimedia.org/v2/envoy-tls-local-proxy/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'

I've not tested the upload status paths, though. For API reference, see: https://docs.docker.com/registry/spec/api/#detail

Also, we are sending a duplicate docker-distribution-api-version: header in most responses.

See also:

Event Timeline

herron triaged this task as Medium priority.Jul 27 2020, 7:36 PM

Change 650153 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/puppet@production] docker_registry_ha: Add "Vary: Accept" to response

https://gerrit.wikimedia.org/r/650153

Change 650153 merged by JMeybohm:
[operations/puppet@production] docker_registry_ha: Add "Vary: Accept" to response

https://gerrit.wikimedia.org/r/650153

Change 691106 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] docker-registry: Clean up old http endpoint

https://gerrit.wikimedia.org/r/691106

Change 691107 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] docker-registry: Remove Docker-Distribution-API-version header

https://gerrit.wikimedia.org/r/691107

Change 691108 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] docker-registry: Re-apply Cache-Control rules

https://gerrit.wikimedia.org/r/691108

Change 691110 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] docker-registry: Remove absented nginx-site resource

https://gerrit.wikimedia.org/r/691110

Change 691106 merged by Alexandros Kosiaris:

[operations/puppet@production] docker-registry: Clean up old nginx http endpoint

https://gerrit.wikimedia.org/r/691106

Change 691107 merged by Alexandros Kosiaris:

[operations/puppet@production] docker-registry: Remove Docker-Distribution-API-version header

https://gerrit.wikimedia.org/r/691107

Change 692275 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] docker-registry: Remove monitoring for port 81

https://gerrit.wikimedia.org/r/692275

Change 692275 merged by Alexandros Kosiaris:

[operations/puppet@production] docker-registry: Remove monitoring for port 81

https://gerrit.wikimedia.org/r/692275

Change 691110 merged by Alexandros Kosiaris:

[operations/puppet@production] docker-registry: Remove absented nginx-site resource

https://gerrit.wikimedia.org/r/691110

Change 691108 merged by Alexandros Kosiaris:

[operations/puppet@production] docker-registry: Re-apply Cache-Control rules

https://gerrit.wikimedia.org/r/691108

Change 692910 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] registry: Fix up

https://gerrit.wikimedia.org/r/692910

Change 692910 merged by Alexandros Kosiaris:

[operations/puppet@production] registry: Add proxy_pass to the catalog location block

https://gerrit.wikimedia.org/r/692910

Change 693430 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] docker_registry_ha: Ensure Vary header is send

https://gerrit.wikimedia.org/r/693430

Change 693430 merged by JMeybohm:

[operations/puppet@production] docker_registry_ha: Ensure Vary header is send

https://gerrit.wikimedia.org/r/693430

Change 694552 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] httpbb: Add tests for docker-registry

https://gerrit.wikimedia.org/r/694552

Change 694552 merged by JMeybohm:

[operations/puppet@production] httpbb: Add tests for docker-registry

https://gerrit.wikimedia.org/r/694552

Change 696403 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] docker_registry_ha: Enable local nginx cache by default

https://gerrit.wikimedia.org/r/696403

Change 696403 merged by JMeybohm:

[operations/puppet@production] docker_registry_ha: Enable local nginx cache by default

https://gerrit.wikimedia.org/r/696403

Change 861463 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] docker_registry_ha: remove unused cache::nodes ref

https://gerrit.wikimedia.org/r/861463

Change 861463 merged by Alexandros Kosiaris:

[operations/puppet@production] docker_registry_ha: remove unused cache::nodes ref

https://gerrit.wikimedia.org/r/861463

@JMeybohm is there anything left to dohere? I think we can resolve.

  • Requests for the catalog are not cached
    • curl -I -XGET 'https://docker-registry.wikimedia.org/v2/_catalog

catalog is now cached.

  • Requests for tag lists are cached
    • curl -I -XGET 'https://docker-registry.wikimedia.org/v2/envoy-tls-local-proxy/tags/list

Still cached, maybe it should not make new tags immediately visible?

  • Requests for manifests are cached (but the cache does not seem to honor the Accept header send by the client T242200)
    • curl -i -XGET -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' 'https://docker-registry.wikimedia.org/v2/envoy-tls-local-proxy/manifests/dontuseme'
    • curl -i -XGET 'https://docker-registry.wikimedia.org/v2/envoy-tls-local-proxy/manifests/sha256:0786b049723e8e9877e0939798dd8f5aaf6a2cd01b194f92d2d5a349e186c335'

They are no longer cached at all

  • HTTP 404 seem to get cached as well
    • curl -i -XGET -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' 'https://docker-registry.wikimedia.org/v2/envoy-tls-local-proxy/manifests/foobar'

not cached, which is ok. But cached without the accept header which is not ok.

  • curl -i -XGET 'https://docker-registry.wikimedia.org/v2/envoy-tls-local-proxy/manifests/sha256:0786b049723e8e9877e0939798dd8f5aaf6a2cd01b194f92d2dccccccccccccc'

cached, probably not ok

  • Requests for blobs are cached (404's here as well)
    • curl -I -XGET 'https://docker-registry.wikimedia.org/v2/envoy-tls-local-proxy/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'

caching 404 here is probably not a good idea as well

I've not tested the upload status paths, though. For API reference, see: https://docs.docker.com/registry/spec/api/#detail

Did not do that this time as well.

Also, we are sending a duplicate docker-distribution-api-version: header in most responses.

This is fixed.

Still lowering priority as we've not seen issues arising from this

JMeybohm lowered the priority of this task from Medium to Low.Mar 8 2024, 2:21 PM