Page MenuHomePhabricator
Create Task
Maniphest T329679

Set up mirror of the docker hub registry for gitlab-runners
Closed, ResolvedPublic
Actions

Description

As demonstrated in T329216 it is easy to unintentionally cause too many requests to be sent to the official docker registry, resulting in failed CI jobs for many users.

Proposal:

  • In all places where gitlab-runners live (wmcs, trusted, releng cloud runners), set up a registry proxy to cache access to the Docker Hub registry.
  • The standard docker registry implementation already has built-in support for this type of proxy caching. https://docs.docker.com/registry/configuration/#proxy
  • This proxy can be equipped with credentials to isolate its Docker Hub pull activity accounting.
  • Configure (or provide advice on how to configure) image-downloading programs to check the mirror before accessing to official registry.
    • buildkitd
      • gitlab-cloud-runners
      • wmcs/trusted runners
    • Kubernetes runtime (using a Pod admission controller) (gitlab-cloud-runners)
    • dockerd
    • ??
  • Block direct access to docker.io registry from all runners This is not necessary if we ensure that the mirror uses its own set of credentials.

Notes:

Docker Hub limits the number of Docker image downloads (“pulls”) based on the account type of the user pulling the image. Pull rates limits are based on individual IP address. For anonymous users, the rate limit is set to 100 pulls per 6 hours per IP address. For authenticated users, it’s 200 pulls per 6 hour period. Users with a paid Docker subscription get up to 5000 pulls per day. If you require a higher number of pulls, you can also purchase an Enhanced Service Account add-on.
...
A pull request is defined as up to two GET requests on registry manifest URLs (/v2/*/manifests/*).
A normal image pull makes a single manifest request.
A pull request for a multi-arch image makes two manifest requests.
HEAD requests aren’t counted.

xref: https://docs.docker.com/docker-hub/download-rate-limit

Details

SubjectRepoBranchLines +/-
gitlab_runner: fix docker run command in registry service, fix hieraoperations/puppetproduction+2 -2
gitlab_runner: add optional docker registry proxy to runnersoperations/puppetproduction+86 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

dancy created this task.Feb 14 2023, 9:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 14 2023, 9:10 PM
dancy added a parent task: T329216: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit".Feb 14 2023, 9:10 PM
Legoktm awarded a token.Feb 14 2023, 9:40 PM
Addshore subscribed.Feb 15 2023, 1:07 PM

In practice I have never used gitlab's registry proxy feature., and it appears that you may need to address the cached images differently?
This could be annoying and a regular registry pull through mirror might be more appealing?

This is fairly easy to run

docker run -d -p 6000:5000 \
    -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
    -e REGISTRY_PROXY_USERNAME=<TODO-USERNAME> \
    -e REGISTRY_PROXY_PASSWORD=<TODO-PASSWORD/KEY> \
    --restart always \
    --name registry registry:2

And configure

# NOTE: If sudo doesn't work for the file change you may need to sudo su, and then run the echo as root...
echo "{\"registry-mirrors\": [\"http://IP:6000\"]}" > /etc/docker/daemon.json
service docker restart
thcipriani moved this task from Backlog to Ready on the Release-Engineering-Team (GitLab V: Event Horizon 🌄) board.Feb 15 2023, 5:56 PM
Isaac subscribed.Feb 15 2023, 8:49 PM
bd808 subscribed.Feb 15 2023, 8:52 PM
This comment was removed by bd808.
bd808 added a comment.Feb 15 2023, 9:18 PM
This comment was removed by bd808.
bd808 mentioned this in T329216: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit".Feb 15 2023, 9:20 PM
fnegri subscribed.Feb 16 2023, 10:28 AM
LSobanski subscribed.Feb 16 2023, 2:32 PM
bd808 added a comment.Feb 16 2023, 4:26 PM

If possible the proxy should be configured to use a named Docker account when pulling from the Docker upstream repo. This will do two things: 1) raise the pull limit for to 200/6 hours, and 2) isolate the proxy's pulls from the default 100/6 hours rate limit on the Cloud VPS egress IP.

dancy added a comment.Feb 16 2023, 4:38 PM

Thanks for the research @bd808 and @Addshore . @bd808 I may hit you up for those credentials soon.

Dzahn renamed this task from Disallow direct access to the docker.io registry from gitlab runners to Disallow direct access to the docker.io registry from gitlab runners (setup a mirror of docker hub).Feb 22 2023, 4:09 PM
Dzahn added a project: collaboration-services.
Dzahn added a project: serviceops-radar.
dancy updated the task description. (Show Details)Feb 22 2023, 6:02 PM
gerritbot added a comment.Feb 22 2023, 6:02 PM

dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/135

Install docker-hub-mirror

gerritbot added a project: Patch-For-Review.Feb 22 2023, 6:02 PM
gerritbot added a comment.Feb 22 2023, 9:13 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/135

Install docker-hub-mirror

Maintenance_bot removed a project: Patch-For-Review.Feb 22 2023, 9:30 PM
Jelto mentioned this in T328283: Cleanup and attach volumes for gitlab-runners WMCS project.Feb 23 2023, 6:41 PM
Jelto moved this task from Incoming to Consultation on the collaboration-services board.Feb 27 2023, 4:56 PM
gerritbot added a comment.Feb 27 2023, 6:18 PM

dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/146

Enable persistent storage for docker-hub-mirror

gerritbot added a project: Patch-For-Review.Feb 27 2023, 6:18 PM
gerritbot added a comment.Feb 27 2023, 6:21 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/146

Enable persistent storage for docker-hub-mirror

Maintenance_bot removed a project: Patch-For-Review.Feb 27 2023, 6:29 PM
dancy moved this task from Ready to In progress on the Release-Engineering-Team (GitLab V: Event Horizon 🌄) board.Feb 27 2023, 8:56 PM
dancy renamed this task from Disallow direct access to the docker.io registry from gitlab runners (setup a mirror of docker hub) to Set up mirror of the docker hub registry for gitlab-runners.Feb 27 2023, 9:11 PM
dancy claimed this task.
dancy triaged this task as Medium priority.
dancy updated the task description. (Show Details)
dancy updated the task description. (Show Details)Feb 27 2023, 9:23 PM
gerritbot added a comment.Mar 2 2023, 3:02 AM

dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/151

Enabled containerd mirror configuration daemonset

gerritbot added a project: Patch-For-Review.Mar 2 2023, 3:02 AM

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/151

Enable containerd mirror configuration daemonset

Maintenance_bot removed a project: Patch-For-Review.Mar 2 2023, 3:10 AM
dancy updated the task description. (Show Details)Mar 2 2023, 3:10 AM
Jelto added a comment.Mar 2 2023, 8:07 AM

One of the WMCS Shared Runners runner-29 has a dedicated disk for storing the mirrored dockerhub images now:

/dev/sdb1       492G   28K  467G   1% /var/lib/docker-registry

I used the unused volumes from T328283 and created a single bigger volume for that. That should give us some space for storing images.

Next step is to add some puppet code to run the registry proxy container. This should be configurable by a feature/ensure flag, as the proxy is not needed on the Trusted Runners and not all WMCS runners.

gerritbot added a comment.Mar 3 2023, 6:22 PM

Change 894100 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: add optional docker registry proxy to runners

https://gerrit.wikimedia.org/r/894100

gerritbot added a project: Patch-For-Review.Mar 3 2023, 6:22 PM
gerritbot added a comment.Mar 8 2023, 10:47 PM

jhuneidi opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/159

Enable persistent storage for docker-hub-mirror

gerritbot added a comment.Mar 8 2023, 10:47 PM

jhuneidi closed https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/159

Enable persistent storage for docker-hub-mirror

gerritbot added a comment.Mar 10 2023, 9:13 AM

Change 894100 merged by Jelto:

[operations/puppet@production] gitlab_runner: add optional docker registry proxy to runners

https://gerrit.wikimedia.org/r/894100

Maintenance_bot removed a project: Patch-For-Review.Mar 10 2023, 9:30 AM
gerritbot added a comment.Mar 10 2023, 10:44 AM

Change 896316 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: fix docker run command in registry service, fix hiera

https://gerrit.wikimedia.org/r/896316

gerritbot added a project: Patch-For-Review.Mar 10 2023, 10:44 AM
gerritbot added a comment.Mar 10 2023, 11:01 AM

Change 896316 merged by Jelto:

[operations/puppet@production] gitlab_runner: fix docker run command in registry service, fix hiera

https://gerrit.wikimedia.org/r/896316

Maintenance_bot removed a project: Patch-For-Review.Mar 10 2023, 11:10 AM
Jelto added a comment.Mar 13 2023, 10:39 AM

A docker registry container is running on runner-1029 in WMCS now. A quick test with enabling the pull-through cache seemed to work. With enabled pull-through cache and a docker pull, the image data ends up in /var/lib/docker-registry.

runner-1029:~$ sudo docker ps
CONTAINER ID   IMAGE        COMMAND                  CREATED          STATUS          PORTS                    NAMES
c56583379804   registry:2   "/entrypoint.sh /etc…"   7 minutes ago    Up 7 minutes    0.0.0.0:5000->5000/tcp   registry

runner-1029:~$ sudo docker pull golang:bullseye

runner-1029:~$ ls /var/lib/docker-registry/
docker  lost+found  scheduler-state.json

runner-1029:~$ cat /var/lib/docker-registry/scheduler-state.json 
{"library/golang@sha256:...":}
...

Before enabling the proxy/pull-through cache for all WMCS runners we need to configure tls properly, so other runners can speak to the proxy over https.

gerritbot added a comment.Mar 16 2023, 6:34 PM

dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/192

docker-hub-mirror: Enable pod admission controller

gerritbot added a project: Patch-For-Review.Mar 16 2023, 6:34 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/192

docker-hub-mirror: Enable pod admission controller

Maintenance_bot removed a project: Patch-For-Review.Mar 16 2023, 7:10 PM
gerritbot added a comment.Mar 16 2023, 10:00 PM

dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/193

docker-hub-mirror: Authenticate with Docker Hub

gerritbot added a project: Patch-For-Review.Mar 16 2023, 10:00 PM
gerritbot added a comment.Mar 17 2023, 5:56 PM

jhuneidi closed https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/191

Move everything to modules, move everything to top-level

gerritbot added a comment.Mar 20 2023, 2:50 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/193

docker-hub-mirror: Authenticate with Docker Hub

dancy updated the task description. (Show Details)Mar 20 2023, 5:02 PM
thcipriani edited projects, added Release-Engineering-Team (They Live 🕶️🧟); removed Release-Engineering-Team (GitLab V: Event Horizon 🌄).May 10 2023, 3:59 PM
jeena moved this task from Backlog to In progress on the Release-Engineering-Team (They Live 🕶️🧟) board.Jun 12 2023, 9:11 PM
dancy closed this task as Resolved.Jul 26 2023, 6:14 PM

I'm resolving this task as "done enough for now". docker-hub-mirror has been deployed to gitlab-cloud-runners along with a pod admisssion webhook and buildkitd configuration and has been working fine there.

dancy moved this task from In progress to Done on the Release-Engineering-Team (They Live 🕶️🧟) board.Jul 26 2023, 7:08 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits