Page MenuHomePhabricator
Create Task
Maniphest T179696

Homepage for https://docker-registry.wikimedia.org
Closed, ResolvedPublic
Actions

Description

I want to be able to see what images exist on the wikimedia docker registry, as well as what tags they have, when they were added etc.

Currently HTTP requests just get 404 page not found

Details

Show related patches Customize query in gerrit

Related Objects

Event Timeline

• Addshore created this task.Nov 3 2017, 3:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 3 2017, 3:01 PM
• Addshore renamed this task from Create UI for https://docker-registry.wikimedia.org to UI for https://docker-registry.wikimedia.org.Nov 3 2017, 3:01 PM
hashar added a subscriber: hashar.Nov 3 2017, 6:21 PM

https://docker-registry.wikimedia.org/v2/_catalog

{
    "repositories": [
        "alpine",
        "calico/kube-policy-controller",
        "calico/node",
        "fluent-bit",
        "fluentd",
        "kubernetes-fluentd-daemonset",
        "nodejs-devel",
        "nodejs-slim",
        "pause",
        "prometheus-statsd-exporter",
        "python3",
        "python3-build-jessie",
        "python3-build-stretch",
        "python3-devel",
        "ruby",
        "servermon",
        "statsd-proxy",
        "wikimedia-jessie",
        "wikimedia-stretch",
        "wikimedia/mediawiki-services-mathoid"
    ]

}

And for a given image/tag:

https://docker-registry.wikimedia.org/v2/wikimedia-jessie/manifests/latest

And there is probably half a dozen of frontends :]

• Addshore renamed this task from UI for https://docker-registry.wikimedia.org to Homepage for https://docker-registry.wikimedia.org.Nov 5 2017, 12:39 PM

I switched the title from UI to Homepage to keep things a little more basic.

I would be happy with a static html page that just linked to the URL / URLS that you just linked in the ticket!

MoritzMuehlenhoff triaged this task as Medium priority.Nov 8 2017, 3:42 PM
hashar removed a subscriber: hashar.Nov 13 2017, 8:04 PM
hashar removed a project: Continuous-Integration-Infrastructure (shipyard).Nov 20 2017, 3:05 PM
• Addshore added a comment.Jan 23 2018, 8:49 PM

Even just a redirect to https://docker-registry.wikimedia.org/v2/_catalog would be nice.

Joe added a comment.Jan 23 2018, 10:32 PM

Portus seems like an interesting approach to two problems: -authn+authz and UI for a docker registry

http://port.us.org/features.html

I'll take a harder look as soon as I have time for it. Maybe during the ops offsite this weekend.

colewhite added a subscriber: colewhite.Aug 28 2018, 9:46 PM

T202504: Evaluate VMWare's Harbour as a docker registry

thcipriani added a subscriber: thcipriani.Dec 18 2018, 11:32 PM

I made a little command line tool to help me find image: https://gist.github.com/thcipriani/7d7633eb238cd868d5ba24d0f1069463

Then I wrapped that in a bash script and generated a simple static site: https://people.wikimedia.org/~thcipriani/docker/

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:34 PM
hashar added a comment.Feb 28 2019, 10:37 PM

And @thcipriani did https://tools.wmflabs.org/dockerregistry/

Lens0021 added a subscriber: Lens0021.Oct 8 2020, 12:57 PM
Michael added a subscriber: Michael.Nov 13 2020, 2:48 PM
dancy added a subscriber: dancy.Dec 17 2020, 4:34 PM

Right now when you visit https://docker-registry.wikimedia.org/ you get a generic nginx welcome page. I propose having this URL redirect to https://dockerregistry.toolforge.org/

gerritbot added a comment.Dec 17 2020, 6:20 PM

Change 650215 had a related patch set uploaded (by Ahmon Dancy; owner: Ahmon Dancy):
[operations/puppet@production] Redirect top level URl to https://dockerregistry.toolforge.org/

https://gerrit.wikimedia.org/r/650215

gerritbot added a project: Patch-For-Review.Dec 17 2020, 6:20 PM
gerritbot added a comment.Jan 7 2021, 1:41 AM

Change 654725 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] [WIP] docker_registry_ha: Add a script to generate a static HTML homepage

https://gerrit.wikimedia.org/r/654725

gerritbot added a comment.Jan 12 2021, 10:55 PM

Change 655792 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] docker_registry_ha: Have nginx serve /srv/hompage for /

https://gerrit.wikimedia.org/r/655792

gerritbot added a comment.Jan 19 2021, 10:40 PM

Change 654725 merged by Legoktm:
[operations/puppet@production] docker_registry_ha: Add a script to generate a static HTML homepage

https://gerrit.wikimedia.org/r/654725

Stashbot added a comment.Jan 20 2021, 12:09 AM

Mentioned in SAL (#wikimedia-operations) [2021-01-20T00:09:25Z] <legoktm> uploaded docker-report 0.0.4-1~deb9u1 to stretch-wikimedia (T179696)

gerritbot added a comment.Jan 20 2021, 12:09 AM

Change 657210 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] docker_registry_ha: Make registery-homepage-builder Python 3.5 compatible

https://gerrit.wikimedia.org/r/657210

gerritbot added a comment.Jan 20 2021, 12:15 AM

Change 657210 merged by Legoktm:
[operations/puppet@production] docker_registry_ha: Make registry-homepage-builder Python 3.5 compatible

https://gerrit.wikimedia.org/r/657210

Legoktm added a subscriber: Legoktm.Jan 20 2021, 12:25 AM

Got pretty close, one last sticking point is that docker_report hardcodes connecting to the registry over HTTPS. So if you try https://localhost then you'll end up with requests.exceptions.SSLError: hostname 'localhost' doesn't match either of 'docker-registry.discovery.wmnet', 'docker-registry.svc.eqiad.wmnet', 'docker-registry.svc.codfw.wmnet', 'docker-registry.wikimedia.org'. And of course https://localhost:5000 (the HTTP port) fails with a protocol error.

Should we adapt docker_report to allow connecting over HTTP? I thought about using one of the domain names but then we're generating a homepage for a different registry, not the one that instance is serving...or does it not matter?

gerritbot added a comment.Jan 20 2021, 12:26 AM

Change 657216 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] docker_registry_ha: Disable build-homepage job for now

https://gerrit.wikimedia.org/r/657216

gerritbot added a comment.Jan 20 2021, 12:29 AM

Change 657216 merged by Legoktm:
[operations/puppet@production] docker_registry_ha: Disable build-homepage job for now

https://gerrit.wikimedia.org/r/657216

Legoktm claimed this task.Jan 20 2021, 12:50 AM
Legoktm added a project: serviceops.
akosiaris added a subscriber: akosiaris.Jan 20 2021, 11:15 AM
In T179696#6760378, @Legoktm wrote:

Got pretty close, one last sticking point is that docker_report hardcodes connecting to the registry over HTTPS. So if you try https://localhost then you'll end up with requests.exceptions.SSLError: hostname 'localhost' doesn't match either of 'docker-registry.discovery.wmnet', 'docker-registry.svc.eqiad.wmnet', 'docker-registry.svc.codfw.wmnet', 'docker-registry.wikimedia.org'. And of course https://localhost:5000 (the HTTP port) fails with a protocol error.

Should we adapt docker_report to allow connecting over HTTP? I thought about using one of the domain names but then we're generating a homepage for a different registry, not the one that instance is serving...or does it not matter?

Or potentially have a flag to not do certificate validation. But we probably want to allow HTTP as well. It should make it easier to test/use it locally as well. Plus, it should make the script a bit faster.

Joe added a comment.EditedJan 20 2021, 11:16 AM
In T179696#6760378, @Legoktm wrote:

Got pretty close, one last sticking point is that docker_report hardcodes connecting to the registry over HTTPS. So if you try https://localhost then you'll end up with requests.exceptions.SSLError: hostname 'localhost' doesn't match either of 'docker-registry.discovery.wmnet', 'docker-registry.svc.eqiad.wmnet', 'docker-registry.svc.codfw.wmnet', 'docker-registry.wikimedia.org'. And of course https://localhost:5000 (the HTTP port) fails with a protocol error.

Should we adapt docker_report to allow connecting over HTTP? I thought about using one of the domain names but then we're generating a homepage for a different registry, not the one that instance is serving...or does it not matter?

Given the banner page we're creating is for use by the public, I think it can simply run against the public registry address. And yes, it should not matter as the storage is shared (it's swift containers).

Joe added a comment.Jan 20 2021, 2:02 PM

For the record, I'm running the script as follows on registry1001:

/usr/local/bin/registry-homepage-builder docker-registry.wikimedia.org /root/homepage-test

to perform a test run, and it seems to work as expected.

Joe added a comment.Jan 20 2021, 4:09 PM

For the record, I got the script running, by using

/usr/local/bin/registry-homepage-builder docker-registry.wikimedia.org /root/homepage-test

as arguments; and then I stumbled upon another problem: fetching all tags for some images gets a consistent 504 timeout (meaning we exceed the timeout of the tls terminator on the registry hosts).

Either we ignore it by just catching requests.exceptions.HTTPError, or we reduce the number of tags we fetch to a smaller number than 100.

Legoktm added a comment.Jan 20 2021, 5:06 PM
In T179696#6761286, @Joe wrote:

Given the banner page we're creating is for use by the public, I think it can simply run against the public registry address. And yes, it should not matter as the storage is shared (it's swift containers).

Ack, sounds good to me.

In T179696#6762296, @Joe wrote:

For the record, I got the script running, by using

/usr/local/bin/registry-homepage-builder docker-registry.wikimedia.org /root/homepage-test

as arguments; and then I stumbled upon another problem: fetching all tags for some images gets a consistent 504 timeout (meaning we exceed the timeout of the tls terminator on the registry hosts).

Either we ignore it by just catching requests.exceptions.HTTPError, or we reduce the number of tags we fetch to a smaller number than 100.

This was fixed in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/docker-report/+/2b57ec7cc2040c34d67b7dd94556f6a1d72ebafe - I can cherry-pick a smaller version of that on top of the 0.0.4 stretch package.

gerritbot added a comment.Jan 20 2021, 9:14 PM

Change 657412 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] docker_registry_ha: Enable and fix build-homepage job

https://gerrit.wikimedia.org/r/657412

gerritbot added a comment.Jan 20 2021, 9:24 PM

Change 657412 merged by Legoktm:
[operations/puppet@production] docker_registry_ha: Enable and fix build-homepage job

https://gerrit.wikimedia.org/r/657412

Legoktm mentioned this in T272550: Upgrade docker-registry servers to Debian Buster.Jan 20 2021, 10:26 PM
gerritbot added a comment.Jan 20 2021, 10:37 PM

Change 655792 merged by Legoktm:
[operations/puppet@production] docker_registry_ha: Have nginx serve /srv/homepage for /

https://gerrit.wikimedia.org/r/655792

Legoktm added a comment.Jan 20 2021, 11:08 PM

https://docker-registry.wikimedia.org/ ta-da

Tested by:

  • docker pull docker-registry.discovery.wmnet/wikimedia-buster:latest on a kubernetes1001
  • docker pulll docker-registry.wikimedia.org/wikimedia-buster:latest on my laptop
  • pushing images to docker-registry.discovery.wmnet from deneb with # DISTRIBUTIONS="buster" build-base-images

It looks like the build-homepage job is failing because of HTTP 504 timeouts though, despite me applying the automatic retry patch to the stretch version of docker-report. Especially when all 4 hosts run the job on the hour at the same time. When just one is running, it most likely will work.

gerritbot added a comment.Jan 20 2021, 11:13 PM

Change 657446 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] docker_registry_ha: Randomize timing of build-homepage job

https://gerrit.wikimedia.org/r/657446

gerritbot added a comment.Jan 20 2021, 11:21 PM

Change 657446 merged by Legoktm:
[operations/puppet@production] docker_registry_ha: Randomize timing of build-homepage job

https://gerrit.wikimedia.org/r/657446

Joe added a comment.Jan 21 2021, 1:47 PM

Macro antoine-approve: {{done}}

Lens0021 awarded a token.Jan 21 2021, 4:06 PM
dancy added a comment.Jan 21 2021, 4:11 PM

Thanks Legoktm. Small feature request: Can you add "last updated at
<blah>" text to the top right corner of the page?

Joe added a comment.Jan 21 2021, 4:26 PM
In T179696#6765834, @dancy wrote:

Thanks Legoktm. Small feature request: Can you add "last updated at
<blah>" text to the top right corner of the page?

+1 that's a very good idea to avoid confusion.

Legoktm added a comment.Jan 21 2021, 7:47 PM
In T179696#6765834, @dancy wrote:

Thanks Legoktm. Small feature request: Can you add "last updated at
<blah>" text to the top right corner of the page?

I'm not the greatest with CSS, but here's how it looks:

Screenshot_2021-01-21 Wikimedia Docker - Image alpine.png (1×3 px, 77 KB)

Screenshot_2021-01-21 Wikimedia Docker - Images.png (1×3 px, 125 KB)

gerritbot added a comment.Jan 21 2021, 7:49 PM

Change 657678 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] docker_registry_ha: Add timestamp to build-homepage output

https://gerrit.wikimedia.org/r/657678

gerritbot added a comment.Jan 22 2021, 7:55 PM

Change 657678 merged by Legoktm:
[operations/puppet@production] docker_registry_ha: Add timestamp to build-homepage output

https://gerrit.wikimedia.org/r/657678

JMeybohm added a subscriber: JMeybohm.Jan 25 2021, 9:41 AM

The job fails on registry2002, leading to icinga alerts

Jan 25 09:30:01 registry2002 systemd[1]: Started Build docker-registry homepage.
Jan 25 09:30:01 registry2002 registry-homepage-builder[31567]: INFO:root:Fetching the image catalog for docker-registry.discovery.wmnet
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]: ERROR:root:Error getting data from the registry
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]: Traceback (most recent call last):
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:   File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 91, in _get_all_pages
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:     resp = self._request(url_part)
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:   File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 82, in _request
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:     response.raise_for_status()
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 893, in raise_for_status
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:     raise HTTPError(http_error_msg, response=self)
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]: requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://docker-registry.discovery.wmnet
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]: Traceback (most recent call last):
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:   File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 91, in _get_all_pages
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:     resp = self._request(url_part)
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:   File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 82, in _request
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:     response.raise_for_status()
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 893, in raise_for_status
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:     raise HTTPError(http_error_msg, response=self)
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]: requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://docker-registry.discovery.wmnet
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]: During handling of the above exception, another exception occurred:
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]: Traceback (most recent call last):
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:   File "/usr/local/bin/registry-homepage-builder", line 136, in <module>
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:     main()
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:   File "/usr/local/bin/registry-homepage-builder", line 122, in main
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:     for image, tags in sorted(registry.get_image_tags(sort=False).items()):
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:   File "/usr/lib/python3/dist-packages/docker_report/registry/browser.py", line 61, in get_image_tags
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:     for image_name in self._get_images_list():
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:   File "/usr/lib/python3/dist-packages/docker_report/registry/browser.py", line 50, in _get_images_list
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:     for resp in self._get_all_pages("/v2/_catalog"):
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:   File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 108, in _get_all_pages
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]:     raise RegistryError(url_part)
Jan 25 09:32:01 registry2002 registry-homepage-builder[31567]: docker_report.registry.RegistryError: /v2/_catalog?last=releng%2Fquibble-jessie-php55&n=100
Jan 25 09:32:01 registry2002 systemd[1]: build-homepage.service: Main process exited, code=exited, status=1/FAILURE
Dzahn added a subscriber: Dzahn.Jan 25 2021, 8:34 PM

Gateway Time-out for url: https://docker-registry.discovery.wmnet

Gotta set HTTP_PROXY/HTTPS_PROXY env variable?

Legoktm added a comment.Jan 25 2021, 9:01 PM
legoktm@registry2002:~$ time curl "https://docker-registry.discovery.wmnet/v2/_catalog?last=releng%2Fquibble-jessie-php55&n=100"
{"repositories":["releng/quibble-jessie-php56","releng/quibble-stretch","releng/quibble-stretch-bundle","releng/quibble-stretch-hhvm","releng/quibble-stretch-php70","releng/quibble-stretch-php71","releng/quibble-stretch-php72","releng/quibble-stretch-php73","releng/quibble-stretch-php74","releng/quibble-stretch-php80","releng/rake","releng/rake-poolcounter","releng/rake-vagrant","releng/release-notes","releng/rust","releng/rust-coverage","releng/shellcheck","releng/sonar-scanner","releng/sury-php","releng/tabs","releng/tox","releng/tox-acme-chief","releng/tox-buster","releng/tox-censorshipmonitoring","releng/tox-cergen","releng/tox-certcentral","releng/tox-conftool","releng/tox-eventlogging","releng/tox-homer","releng/tox-labs-striker","releng/tox-mysqld","releng/tox-poolcounter","releng/tox-pyspark","releng/tox-pywikibot","releng/typos","releng/wikimedia-audit-resources","releng/zuul-cloner","ruby","servermon","service-checker","statsd-proxy","tiller","wikimedia/blubber","wikimedia/blubber-doc-example-helloworldoid","wikimedia/eventgate-ci","wikimedia/eventgate-wikimedia","wikimedia/mediawiki-core","wikimedia/mediawiki-libs-shellbox","wikimedia/mediawiki-services-apertium","wikimedia/mediawiki-services-change-propagation","wikimedia/mediawiki-services-chromium-render","wikimedia/mediawiki-services-citoid","wikimedia/mediawiki-services-cxserver","wikimedia/mediawiki-services-eventstreams","wikimedia/mediawiki-services-graphoid","wikimedia/mediawiki-services-kask","wikimedia/mediawiki-services-mathoid","wikimedia/mediawiki-services-mobileapps","wikimedia/mediawiki-services-parsoid","wikimedia/mediawiki-services-push-notifications","wikimedia/mediawiki-services-recommendation-api","wikimedia/mediawiki-services-restbase","wikimedia/mediawiki-services-similar-users","wikimedia/mediawiki-services-wikifeeds","wikimedia/mediawiki-services-wikispeech-mary-tts","wikimedia/mediawiki-services-wikispeech-mishkal","wikimedia/mediawiki-services-wikispeech-pronlex","wikimedia/mediawiki-services-wikispeech-symbolset","wikimedia/mediawiki-services-wikispeech-wikispeech-server","wikimedia/mediawiki-services-zotero","wikimedia/research-mwaddlink","wikimedia/wikibase-termbox","wikimedia/wikidata-query-flink-rdf-streaming-updater","wikimedia/wikimedia-portals","wikimedia/wikimedia-production","wikimedia/wvui","wikimedia-buster","wikimedia-jessie","wikimedia-stretch","wmfdebug"]}

real	0m34.496s
user	0m0.016s
sys	0m0.012s

docker-report retries this request 3 times, and it's failing for each time. I tried it a few times with curl manually and it works fine... I don't understand what's different about registry2002 that its requests fail but the other 3 hosts are perfectly OK.

Legoktm added a comment.Jan 25 2021, 9:07 PM
legoktm@registry2002:~$ time curl "https://docker-registry.discovery.wmnet/v2/_catalog?last=releng%2Fquibble-jessie-php55&n=100"
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.13.6</center>
</body>
</html>

real	1m0.036s
user	0m0.028s
sys	0m0.004s

We could also increase the timeout on the nginx side given that every client is going to retry the request anyways?

Dzahn added a comment.Jan 25 2021, 9:08 PM

Fwiw, i get the same timeout when doing that curl command from registry1002.

gerritbot added a comment.Jan 25 2021, 9:54 PM

Change 658436 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] docker_registry_ha: Increase nginx proxy timeout to 120s

https://gerrit.wikimedia.org/r/658436

Legoktm added a comment.Jan 25 2021, 9:56 PM

In my testing of repeatedly issuing the same curl command over and over, it usually took ~35s to respond, but sometimes it took over 1m, the worst I saw was 1m42s. So I'm proposing to raise the nginx proxy timeout to 2m to avoid needing to retry on these timeouts in the first place.

Joe added a comment.Jan 26 2021, 7:04 AM
In T179696#6775314, @Legoktm wrote:

In my testing of repeatedly issuing the same curl command over and over, it usually took ~35s to respond, but sometimes it took over 1m, the worst I saw was 1m42s. So I'm proposing to raise the nginx proxy timeout to 2m to avoid needing to retry on these timeouts in the first place.

I think the nginx timeout can be raised without too much concern here.

gerritbot added a comment.Jan 26 2021, 7:06 AM

Change 658436 merged by Giuseppe Lavagetto:
[operations/puppet@production] docker_registry_ha: Increase nginx proxy timeout to 120s

https://gerrit.wikimedia.org/r/658436

akosiaris added a comment.Jan 26 2021, 9:19 AM
In T179696#6776337, @Joe wrote:
In T179696#6775314, @Legoktm wrote:

In my testing of repeatedly issuing the same curl command over and over, it usually took ~35s to respond, but sometimes it took over 1m, the worst I saw was 1m42s. So I'm proposing to raise the nginx proxy timeout to 2m to avoid needing to retry on these timeouts in the first place.

I think the nginx timeout can be raised without too much concern here.

Alternatively, bypass nginx and talk HTTP directly to the registry software. It's anyway on the same host. It will require some modification of the software, but it will avoid such issues.

Joe added a comment.Jan 26 2021, 10:23 AM

we're still experiencing timeouts when trying to gather the catalog list with the url
/v2/_catalog?last=releng%2Fquibble-jessie-php55&n=100

this only happens on registry2002 consistently.

Interestingly, it seems like requesting the url multiple times makes it respond in ~ 40 seconds on retries, so it's a bit strange we're still failing.

Joe added a comment.Jan 26 2021, 11:12 AM

Looking at the logs from a failed run, it looks like no retry is attempted when a 504 is received, at least on registry2002.

Every 504 from the registry is not followed by another call that ends in a 200 or otherwise.

Legoktm added a comment.Jan 26 2021, 9:34 PM
In T179696#6776832, @Joe wrote:

Looking at the logs from a failed run, it looks like no retry is attempted when a 504 is received, at least on registry2002.

Every 504 from the registry is not followed by another call that ends in a 200 or otherwise.

I see it retrying 3 times:

Jan 26 20:30:02 registry2002 registry-homepage-builder[22695]: INFO:root:Fetching the image catalog for docker-registry.discovery.wmnet
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]: ERROR:root:Error getting data from the registry
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]: Traceback (most recent call last):
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:   File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 91, in _get_all_pages
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:     resp = self._request(url_part)
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:   File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 82, in _request
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:     response.raise_for_status()
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 893, in raise_for_status
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:     raise HTTPError(http_error_msg, response=self)
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]: requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://docker-registry.discovery.wmnet/v2/_catalog?last=releng%2Fquibble-jessie-php55&n=100
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]: Traceback (most recent call last):
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:   File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 91, in _get_all_pages
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:     resp = self._request(url_part)
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:   File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 82, in _request
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:     response.raise_for_status()
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:   File "/usr/lib/python3/dist-packages/requests/models.py", line 893, in raise_for_status
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:     raise HTTPError(http_error_msg, response=self)
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]: requests.exceptions.HTTPError: 504 Server Error: Gateway Time-out for url: https://docker-registry.discovery.wmnet/v2/_catalog?last=releng%2Fquibble-jessie-php55&n=100
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]: During handling of the above exception, another exception occurred:
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]: Traceback (most recent call last):
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:   File "/usr/local/bin/registry-homepage-builder", line 136, in <module>
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:     main()
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:   File "/usr/local/bin/registry-homepage-builder", line 122, in main
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:     for image, tags in sorted(registry.get_image_tags(sort=False).items()):
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:   File "/usr/lib/python3/dist-packages/docker_report/registry/browser.py", line 61, in get_image_tags
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:     for image_name in self._get_images_list():
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:   File "/usr/lib/python3/dist-packages/docker_report/registry/browser.py", line 50, in _get_images_list
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:     for resp in self._get_all_pages("/v2/_catalog"):
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:   File "/usr/lib/python3/dist-packages/docker_report/registry/__init__.py", line 108, in _get_all_pages
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]:     raise RegistryError(url_part)
Jan 26 20:32:13 registry2002 registry-homepage-builder[22695]: docker_report.registry.RegistryError: /v2/_catalog?last=releng%2Fquibble-jessie-php55&n=100
In T179696#6776531, @akosiaris wrote:

Alternatively, bypass nginx and talk HTTP directly to the registry software. It's anyway on the same host. It will require some modification of the software, but it will avoid such issues.

This seems like the most foolproof thing to do, so I'll submit patches, but I still am curious why this is going wrong in this specific case only on registry2002.

gerritbot added a comment.Jan 26 2021, 9:50 PM

Change 658684 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/docker-images/docker-report@master] Allow talking to the registry over HTTP

https://gerrit.wikimedia.org/r/658684

gerritbot added a comment.Jan 27 2021, 10:34 PM

Change 658684 merged by jenkins-bot:
[operations/docker-images/docker-report@master] Allow talking to the registry over HTTP

https://gerrit.wikimedia.org/r/658684

gerritbot added a comment.Jan 27 2021, 11:10 PM

Change 659095 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] docker_registry_ha: Have build-homepage talk directly to the registry

https://gerrit.wikimedia.org/r/659095

gerritbot added a comment.Jan 27 2021, 11:16 PM

Change 659095 merged by Legoktm:
[operations/puppet@production] docker_registry_ha: Have build-homepage talk directly to the registry

https://gerrit.wikimedia.org/r/659095

gerritbot added a comment.Jan 29 2021, 4:24 PM

Change 650215 abandoned by Ahmon Dancy:
[operations/puppet@production] Redirect top level URL to https://dockerregistry.toolforge.org/

Reason:
superseded by https://gerrit.wikimedia.org/r/c/operations/puppet/ /654725

https://gerrit.wikimedia.org/r/650215

Legoktm closed this task as Resolved.Mar 2 2021, 9:52 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL