Dockerhub rate limiting a potential problem for dev environment development
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jeena
	Aug 6 2021, 8:56 PM

Description

The docker development environment in mw cli relies on several images from dockerhub: adminer, defreitas/dns-proxy-server, jwilder/nginx-proxy, graphiteapp/graphite-statsd, maraidb, mysql, postgres, phpmyadmin, redis

Pulling these images in CI (currently run in github) every time a patch is uploaded could cause the rate limits to be hit and slow down the development process.

We need to figure out how to avoid the rate limits, whether it's hosting our own images or figuring out how to not always pull an image if it can be stored in a local docker registry for the next CI run.

Details

	Reference	Source Branch	Dest Branch	Author	Title
	repos/releng/cli!17	gitlab-ci-no-specified-mirror	main	addshore	.gitlab-ci.yml: Don't specify docker mirror here

Customize query in GitLab

Related Objects

Mentioned Here: T329216: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"

Event Timeline

jeena created this task.Aug 6 2021, 8:56 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 6 2021, 8:56 PM

jeena moved this task from Inbox to Discuss & Decide on the mwcli board.Aug 6 2021, 9:21 PM

For now I have mainly got around this by adding the google cloud public mirror for docker hub that has no limits
https://gitlab.wikimedia.org/releng/cli/-/blob/d5c5788711bfa6b5166b71bae97c574ecbe0415a/.gitlab-ci.yml#L39

I also wrote a job that will check the quota for the runner that mwcli is currently using

https://gitlab.wikimedia.org/addshore/test/-/jobs/764

We might end up encountering this again at some point, but for now everything is OK!

Restricted Application added a project: User-Addshore. · View Herald TranscriptSep 9 2021, 2:36 PM

Of course moments after closing this it happened

https://gitlab.wikimedia.org/releng/cli/-/jobs/1093

toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Today was quite a heavy day for development etc..

I'm going to experiment with a docker registry mirror running on the CI runner

https://gitlab.wikimedia.org/addshore/test/-/commit/a70793f4c62fbc6a8e2f6354f79c30a19f8ef3a7

Mainly from https://about.gitlab.com/blog/2020/10/30/mitigating-the-impact-of-docker-hub-pull-requests-limits/

Create a mirror (using docker):

sh
sudo docker run -d -p 6000:5000 \
    -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
    --restart always \
    --name registry registry:2

Get the IP address:

ssh
hostname --ip-address

Add the mirror:

sh
sudo service docker stop
sudo echo '{"registry-mirrors": ["http://172.16.7.26:6000"]}' > /etc/docker/daemon.json
sudo service docker restart

• Addshore moved this task from Discuss & Decide to In Progress on the mwcli board.Sep 9 2021, 4:09 PM

I have "pounded" CI today after setting this up and had no issues.

And even after "all of the CI" most of the limit remains

ratelimit-limit: 100;w=21600
ratelimit-remaining: 99;w=21600

Cross linking to T329216: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit"

Dockerhub rate limiting a potential problem for dev environment developmentClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

Dockerhub rate limiting a potential problem for dev environment development
Closed, ResolvedPublic
Actions