There have been a few times I've had to clean old Docker images off of contint1001 because space on the root partition was getting tight.
Docker uses /var/lib/docker to store its images and layer cache, but partition mounted at / is only 50GB in size. It's currently easier than it should be to fill our CI master with Docker images.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Maintenance: cleanup contint1001, too | integration/config | master | +35 -18 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Dzahn | T178663 Switch CI Docker Storage Driver to its own partition and to use devicemapper | |||
| Invalid | None | T207702 contint1001:/var/lib/docker growth | |||
| Resolved | Joe | T207703 Pruning docker-pkg images | |||
| Resolved | hashar | T207707 contint1001 store docker images on separate partition or disk | |||
| Resolved | thcipriani | T217094 disk space alert on contint1001 | |||
| Resolved | • dduvall | T219850 contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): | |||
| Unknown Object (Task) |
it seems is almost full again. Did you consider to set up a periodic docker image cleanup? cc @hashar
a naive straightforward approach will be something like executing docker image prune -a --force --filter "until=240h" (this works on 18.09 which is installed on contint1001).
We cant really do that since docker-pkg relies on those images being there. Though I have added support for it to automatically pull images from the registry if they are not present, last time I checked that version was not deployed. There is also a prune command, but it only keep the latest version of the images and still delete images that might still be used. So it is not perfect yet :]
There are are also images being build for mediawiki services which take their share of space.
Anyway, the fault is that Docker images are on the / partition and they should be moved to a dedicated (and larger) partition, potentially with additional disks. That is the subject of T207707
Change 519083 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[integration/config@master] maintentance: cleanup contint1001, too
This seems a duplicate of T207707 and see T207707#5315360 for progress there. The image dir has actually been moved to new disks now.
Well maybe not duplicate since at some point growth would of course also affect the new storage location.
Change 519083 merged by jenkins-bot:
[integration/config@master] Maintenance: cleanup contint1001, too