On contint1001 Docker uses the overlay engine on the root partition (/var/lib/docker/overlay2). That should be moved to a dedicated LVS logical volume and the backend switched to devicemapper.
from docker info:
Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true
If I remember properly, we should switch to devicemapper.
We have installed Docker via the upstream Debian package 17.06.2~ce-0~debian. It seems to select overlay2 by default.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Dzahn | T178663 Switch CI Docker Storage Driver to its own partition and to use devicemapper | |||
| Resolved | hashar | T207707 contint1001 store docker images on separate partition or disk | |||
| Resolved | thcipriani | T217094 disk space alert on contint1001 | |||
| Resolved | • dduvall | T219850 contint1001: DISK WARNING - free space: /srv 88397 MB (10% inode=94%): | |||
| Unknown Object (Task) |
Looks like profile::docker::storage has the logic to setup a partition with parameters:
# list of physical volumes to use.
$physical_volumes = hiera('profile::docker::storage::physical_volumes'),
# Volume group to substitute.
$vg_to_remove = hiera('profile::docker::storage::vg_to_remove'),It seems to create a new volume group docker with logical volumes data and metadata.
profile::docker::storage::physical_volumes would be the physical volume.
contint1001 has a 1TB disk and all the physical volume / volume group is allocated:
# pvdisplay --- Physical volume --- PV Name /dev/md2 VG Name contint1001-vg PV Size 883.89 GiB / not usable 3.00 MiB Allocatable yes (but full)
# vgdisplay --- Volume group --- VG Name contint1001-vg Format lvm2 VG Status resizable Cur LV 1 VG Size 883.89 GiB
root@contint1001:~# lvdisplay --- Logical volume --- LV Path /dev/contint1001-vg/data LV Name data VG Name contint1001-vg LV Size 883.89 GiB
Seems to me we would have to shrink the logical volume /dev/contint1001-vg/data and the volume group contint1001-vg.
Mentioned in SAL (#wikimedia-operations) [2018-03-21T08:09:34Z] <hashar> contint1001: docker image prune ; docker container prune # T178663
contint1001 was close to running out of disk space today, Icinga alerted us.
Turns out it was due to docker images being rebuilt and it uses /var/lib/docker on / which has 46G of which 96% were in use.
At the same time contint1001 has a separate mount on /srv with over 400G free that is used only 51%.
In puppet the /var/lib/docker path appears in:
modules/profile/manifests/docker/storage/loopback.pp:# [*dm_source*] Source device for the /var/lib/docker directory
modules/profile/manifests/docker/storage/loopback.pp: $dm_target = '/var/lib/docker'
Can /srv/ be used instead for this?
From my previous comment T178663#3699074 , I would love Docker material to be on a standalone partition. Namely shrink /srv somehow and create a new one. Then we can probably change the Docker storage driver to devicemapper and docker pull all the images we need from the registry.
Added T207707 as a subtask since it is about getting additional storage space without specifying any movement of the storage driver.
We don't want to resize /srv/ as that's already in use for zuul-merger (and is currently using > half the available disk space). Ideally we'd be able to add a disk to this machine just to store images.
See progress on T207707 . The new disks are mounted now and /mnt/docker can be used.
But per T207707#5304252 i would say we should NOT use devicemapper but stick to overlay2.
So i think one part of this ticket is duplicate of T207707 and the other part should be declined which would mean we can close this.