Page MenuHomePhabricator

Dockerhub rate limiting a potential problem for dev environment development
Closed, ResolvedPublic

Description

The docker development environment in mw cli relies on several images from dockerhub: adminer, defreitas/dns-proxy-server, jwilder/nginx-proxy, graphiteapp/graphite-statsd, maraidb, mysql, postgres, phpmyadmin, redis

Pulling these images in CI (currently run in github) every time a patch is uploaded could cause the rate limits to be hit and slow down the development process.

We need to figure out how to avoid the rate limits, whether it's hosting our own images or figuring out how to not always pull an image if it can be stored in a local docker registry for the next CI run.

Event Timeline

For now I have mainly got around this by adding the google cloud public mirror for docker hub that has no limits
https://gitlab.wikimedia.org/releng/cli/-/blob/d5c5788711bfa6b5166b71bae97c574ecbe0415a/.gitlab-ci.yml#L39

I also wrote a job that will check the quota for the runner that mwcli is currently using

https://gitlab.wikimedia.org/addshore/test/-/jobs/764

Addshore claimed this task.

We might end up encountering this again at some point, but for now everything is OK!

Addshore reopened this task as Open.EditedSep 9 2021, 3:54 PM

Of course moments after closing this it happened

https://gitlab.wikimedia.org/releng/cli/-/jobs/1093

toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Today was quite a heavy day for development etc..

I'm going to experiment with a docker registry mirror running on the CI runner

https://gitlab.wikimedia.org/addshore/test/-/commit/a70793f4c62fbc6a8e2f6354f79c30a19f8ef3a7

Mainly from https://about.gitlab.com/blog/2020/10/30/mitigating-the-impact-of-docker-hub-pull-requests-limits/

Create a mirror (using docker):

sh
sudo docker run -d -p 6000:5000 \
    -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
    --restart always \
    --name registry registry:2

Get the IP address:

ssh
hostname --ip-address

Add the mirror:

sh
sudo service docker stop
sudo echo '{"registry-mirrors": ["http://172.16.7.26:6000"]}' > /etc/docker/daemon.json
sudo service docker restart

I have "pounded" CI today after setting this up and had no issues.

And even after "all of the CI" most of the limit remains

ratelimit-limit: 100;w=21600
ratelimit-remaining: 99;w=21600