Page MenuHomePhabricator

Clean up old Docker images on deneb
Open, MediumPublic

Description

/var/lib/docker on deneb is > 70G.

Event Timeline

Legoktm triaged this task as Unbreak Now! priority.Jul 23 2021, 1:04 AM
Legoktm created this task.

user homes over 1G:

1.4G	ema
2.6G	jbond
5.2G	jmm
5.9G	filippo
7.7G	razzi
7.9G	akosiaris
8.6G	elukey
41G	otto

Please see if something can be cleaned up

70G in /var/lib/docker...we have some pretty old images that can also be cleaned up from there

Mentioned in SAL (#wikimedia-operations) [2021-07-23T01:20:57Z] <legoktm> legoktm@deneb:~$ docker rmi docker-registry.wikimedia.org/mwcachedir:0.0.1 # T287222

Mentioned in SAL (#wikimedia-operations) [2021-07-23T01:20:57Z] <legoktm> legoktm@deneb:~$ docker rmi docker-registry.wikimedia.org/mwcachedir:0.0.1 # T287222

Freed up 6G. It should be saved in the registry in case anyone was using it.

Thanks for the ping! Freed stuff on my home dir:

elukey@deneb:~$ du -hs
136M	.
elukey lowered the priority of this task from Unbreak Now! to High.Jul 23 2021, 6:51 AM

Lowering down priority since we have now 22G available.

Cleared my home down to 1.2G. /home in total is down to 30G (and Alex is out), I'm retitling the task to trim the Docker data.

Mentioned in SAL (#wikimedia-operations) [2021-07-26T07:17:47Z] <_joe_> manage-production-images prune on deneb, T287222

MoritzMuehlenhoff renamed this task from deneb.codfw.wmnet root partition is full to Clean up old Docker images on deneb.Aug 2 2021, 7:02 AM
MoritzMuehlenhoff lowered the priority of this task from High to Medium.

I've done some cleaning in my home too, down to ~500M now.

Cleaned up some old istio/knative/kubeflow images, got down to this:

Data Space Used: 66.39GB
Data Space Total: 107.4GB
Data Space Available: 40.98GB

We should try to reduce the images even more, I got a build error today from docker-pkg stating:

[docker-pkg-build] ERROR - Build failed: devmapper: Thin Pool has 158763 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior (image.py:205)

Failed to build an image today:

2021-09-13 09:51:42,980 [docker-pkg-build] ERROR - Build failed: devmapper: Thin Pool has 149221 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior (image.py:205)
/dev/vda1       230G  166G   53G  76% /

I ran prune-production-images.service which was enough for my current needs:

/dev/vda1       230G  140G   78G  65% /

We also have a bunch of releng images there and they won't be cleaned up AIUI:

docker-registry.wikimedia.org/releng/composer-scratch                    1.10.22             645603d0c871        4 months ago        2MB
docker-registry.wikimedia.org/releng/ci-common                           0.4                 8dc9644f1942        6 months ago        807B
docker-registry.wikimedia.org/releng/maven                               3.5.2-1             09ea7f38823f        11 months ago       10.6MB
docker-registry.wikimedia.org/releng/npm-test-graphoid                   0.3.0-s2            9411a3651342        12 months ago       440MB
docker-registry.wikimedia.org/releng/npm-test-3d2png                     0.3.0-s2            9d74588b3ebf        12 months ago       494MB
docker-registry.wikimedia.org/releng/npm-test                            0.7.1-s1            1e032858f949        12 months ago       393MB
docker-registry.wikimedia.org/releng/npm                                 0.4.0-s1            37326b61fe59        12 months ago       393MB
docker-registry.wikimedia.org/releng/ci-jessie                           0.5.1-s1            6d297ee12d71        12 months ago       159MB
docker-registry.wikimedia.org/releng/composer-test-php56                 0.2.0-s2            e72dedfabcdd        16 months ago       322MB
docker-registry.wikimedia.org/releng/composer-php56                      0.2.0-s2            84a4c02ec933        16 months ago       322MB
docker-registry.wikimedia.org/releng/php56                               0.1.2               62335a14b59f        17 months ago       319MB
docker-registry.wikimedia.org/releng/ci-src-setup                        0.3.1-s3            0360e06f7692        23 months ago       326MB
docker-registry.wikimedia.org/releng/npm-test-librdkafka                 0.2.1-s1            0859dde0466b        2 years ago         400MB
# docker system df

TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              194                 17                  52.23GB             51.16GB (97%)
Containers          31                  0                   9.356GB             9.356GB (100%)
Local Volumes       3235                0                   0B                  0B
Build Cache         0                   0                   0B                  0B

What about we just run docker system prune --force from time to time? I mean that would force images to be pulled from the registry again on rebuilds, but that might not be such a big deal in terms of build time.