Page MenuHomePhabricator

contint1001:/var/lib/docker growth
Closed, InvalidPublic


There have been a few times I've had to clean old Docker images off of contint1001 because space on the root partition was getting tight.

Docker uses /var/lib/docker to store its images and layer cache, but partition mounted at / is only 50GB in size. It's currently easier than it should be to fill our CI master with Docker images.


Related Gerrit Patches:
integration/config : masterMaintenance: cleanup contint1001, too

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 22 2018, 9:02 PM
thcipriani triaged this task as Medium priority.Dec 18 2018, 5:33 PM
thcipriani moved this task from Backlog to CI on the Release Pipeline board.
fsero added a subscriber: fsero.Apr 15 2019, 2:58 PM

it seems is almost full again. Did you consider to set up a periodic docker image cleanup? cc @hashar

a naive straightforward approach will be something like executing docker image prune -a --force --filter "until=240h" (this works on 18.09 which is installed on contint1001).

We cant really do that since docker-pkg relies on those images being there. Though I have added support for it to automatically pull images from the registry if they are not present, last time I checked that version was not deployed. There is also a prune command, but it only keep the latest version of the images and still delete images that might still be used. So it is not perfect yet :]

There are are also images being build for mediawiki services which take their share of space.

Anyway, the fault is that Docker images are on the / partition and they should be moved to a dedicated (and larger) partition, potentially with additional disks. That is the subject of T207707

Change 519083 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[integration/config@master] maintentance: cleanup contint1001, too

Dzahn added a comment.Jul 8 2019, 10:18 PM

This seems a duplicate of T207707 and see T207707#5315360 for progress there. The image dir has actually been moved to new disks now.

Dzahn added a comment.Jul 8 2019, 10:19 PM

Well maybe not duplicate since at some point growth would of course also affect the new storage location.

Change 519083 merged by jenkins-bot:
[integration/config@master] Maintenance: cleanup contint1001, too

thcipriani closed this task as Invalid.Jul 16 2019, 4:56 PM

We use /mnt/docker now and it's got a lot of space.