Page MenuHomePhabricator

contint1001:/var/lib/docker growth
Open, NormalPublic

Description

There have been a few times I've had to clean old Docker images off of contint1001 because space on the root partition was getting tight.

Docker uses /var/lib/docker to store its images and layer cache, but partition mounted at / is only 50GB in size. It's currently easier than it should be to fill our CI master with Docker images.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 22 2018, 9:02 PM
thcipriani triaged this task as Normal priority.Dec 18 2018, 5:33 PM
thcipriani moved this task from Backlog to CI on the Release Pipeline board.
fsero added a subscriber: fsero.Apr 15 2019, 2:58 PM

it seems is almost full again. Did you consider to set up a periodic docker image cleanup? cc @hashar

a naive straightforward approach will be something like executing docker image prune -a --force --filter "until=240h" (this works on 18.09 which is installed on contint1001).

We cant really do that since docker-pkg relies on those images being there. Though I have added support for it to automatically pull images from the registry if they are not present, last time I checked that version was not deployed. There is also a prune command, but it only keep the latest version of the images and still delete images that might still be used. So it is not perfect yet :]

There are are also images being build for mediawiki services which take their share of space.

Anyway, the fault is that Docker images are on the / partition and they should be moved to a dedicated (and larger) partition, potentially with additional disks. That is the subject of T207707