Automatically prune docker to clear disk space on deneb.codfw.wmnet
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• razzi
	Mar 24 2022, 6:47 PM

Description

Noticed this alert:

docker system prune freed up 32GB:

razzi@deneb:~$ sudo docker system prune
WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all dangling images
        - all dangling build cache
Are you sure you want to continue? [y/N] y
...
Total reclaimed space: 33.22GB

Consensus in #wikimedia-sre is to do docker system prune periodically like it happens on CI.

Site note: there are also some user directories taking up a decent amount of space (shown below is >500MB):

razzi@deneb:/home$ sudo du -sh * | sort -h
...
607M    razzi
718M    jiji
758M    ema
790M    cwhite
832M    jmm
1015M   dzahn
1.1G    btullis
1.3G    oblivian
1.4G    legoktm
2.4G    jbond
5.9G    filippo
9.2G    akosiaris
17G     otto

Details

	Project	Branch	Lines +/-	Subject
	operations/puppet	production	+31 -50	docker: move pruning to new profile docker::prune
	operations/puppet	production	+26 -0	package_builder: run docker prune on a timer

Customize query in gerrit

Related Objects

Mentioned In: T287222: Clean up old Docker images on deneb

Event Timeline

• razzi created this task.Mar 24 2022, 6:47 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 24 2022, 6:47 PM

Change 773622 had a related patch set uploaded (by Razzi; author: Razzi):

[operations/puppet@production] package_builder: run docker prune on a timer

https://gerrit.wikimedia.org/r/773622

gerritbot added a project: Patch-For-Review.Mar 24 2022, 6:48 PM

Change 773622 merged by Razzi:

[operations/puppet@production] package_builder: run docker prune on a timer

https://gerrit.wikimedia.org/r/773622

Timers are present!

razzi@deneb:~$ systemctl list-timers | grep docker
...
Fri 2022-03-25 03:58:40 UTC  8h left             n/a                          n/a                docker-system-prune-dangling.timer              docker-system-prune-dangling.service
Sun 2022-03-27 03:34:49 UTC  2 days left         n/a                          n/a                docker-system-prune-all.timer                   docker-system-prune-all.service
...

Maintenance_bot removed a project: Patch-For-Review.Mar 24 2022, 7:10 PM

Change 773641 had a related patch set uploaded (by Razzi; author: Razzi):

[operations/puppet@production] docker: move pruning to new profile docker::prune

https://gerrit.wikimedia.org/r/773641

gerritbot added a project: Patch-For-Review.Mar 24 2022, 9:27 PM

JMeybohm mentioned this in T287222: Clean up old Docker images on deneb.Mar 25 2022, 8:29 AM

Change 773641 merged by Razzi: