Page MenuHomePhabricator

contint1001:/var/lib/docker growth
Closed, InvalidPublic


There have been a few times I've had to clean old Docker images off of contint1001 because space on the root partition was getting tight.

Docker uses /var/lib/docker to store its images and layer cache, but partition mounted at / is only 50GB in size. It's currently easier than it should be to fill our CI master with Docker images.

Event Timeline

thcipriani triaged this task as Medium priority.Dec 18 2018, 5:33 PM
thcipriani moved this task from Backlog to CI on the Release Pipeline board.

it seems is almost full again. Did you consider to set up a periodic docker image cleanup? cc @hashar

a naive straightforward approach will be something like executing docker image prune -a --force --filter "until=240h" (this works on 18.09 which is installed on contint1001).

We cant really do that since docker-pkg relies on those images being there. Though I have added support for it to automatically pull images from the registry if they are not present, last time I checked that version was not deployed. There is also a prune command, but it only keep the latest version of the images and still delete images that might still be used. So it is not perfect yet :]

There are are also images being build for mediawiki services which take their share of space.

Anyway, the fault is that Docker images are on the / partition and they should be moved to a dedicated (and larger) partition, potentially with additional disks. That is the subject of T207707

Change 519083 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[integration/config@master] maintentance: cleanup contint1001, too

This seems a duplicate of T207707 and see T207707#5315360 for progress there. The image dir has actually been moved to new disks now.

Well maybe not duplicate since at some point growth would of course also affect the new storage location.

Change 519083 merged by jenkins-bot:
[integration/config@master] Maintenance: cleanup contint1001, too

We use /mnt/docker now and it's got a lot of space.