Page MenuHomePhabricator

Switch CI Docker Storage Driver to its own partition and to use devicemapper
Closed, ResolvedPublic


On contint1001 Docker uses the overlay engine on the root partition (/var/lib/docker/overlay2). That should be moved to a dedicated LVS logical volume and the backend switched to devicemapper.

from docker info:

Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true

If I remember properly, we should switch to devicemapper.

We have installed Docker via the upstream Debian package 17.06.2~ce-0~debian. It seems to select overlay2 by default.

Event Timeline

Looks like profile::docker::storage has the logic to setup a partition with parameters:

# list of physical volumes to use.
$physical_volumes = hiera('profile::docker::storage::physical_volumes'),
# Volume group to substitute.
$vg_to_remove = hiera('profile::docker::storage::vg_to_remove'),

It seems to create a new volume group docker with logical volumes data and metadata.

profile::docker::storage::physical_volumes would be the physical volume.

contint1001 has a 1TB disk and all the physical volume / volume group is allocated:

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/md2
  VG Name               contint1001-vg
  PV Size               883.89 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
# vgdisplay 
  --- Volume group ---
  VG Name               contint1001-vg
  Format                lvm2
  VG Status             resizable
  Cur LV                1
  VG Size               883.89 GiB
root@contint1001:~# lvdisplay 
  --- Logical volume ---
  LV Path                /dev/contint1001-vg/data
  LV Name                data
  VG Name                contint1001-vg
  LV Size                883.89 GiB

Seems to me we would have to shrink the logical volume /dev/contint1001-vg/data and the volume group contint1001-vg.

low priority since right now overlay2 does not seem to be an issue.

hashar renamed this task from Switch CI Docker Storage Driver to devicemapper to Switch CI Docker Storage Driver to its own partition and to use devicemapper.Mar 21 2018, 8:07 AM
hashar updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2018-03-21T08:09:34Z] <hashar> contint1001: docker image prune ; docker container prune # T178663

contint1001 was close to running out of disk space today, Icinga alerted us.

Turns out it was due to docker images being rebuilt and it uses /var/lib/docker on / which has 46G of which 96% were in use.

At the same time contint1001 has a separate mount on /srv with over 400G free that is used only 51%.

In puppet the /var/lib/docker path appears in:

modules/profile/manifests/docker/storage/loopback.pp:# [*dm_source*] Source device for the /var/lib/docker directory
modules/profile/manifests/docker/storage/loopback.pp: $dm_target = '/var/lib/docker'

Can /srv/ be used instead for this?

From my previous comment T178663#3699074 , I would love Docker material to be on a standalone partition. Namely shrink /srv somehow and create a new one. Then we can probably change the Docker storage driver to devicemapper and docker pull all the images we need from the registry.

Added T207707 as a subtask since it is about getting additional storage space without specifying any movement of the storage driver.

We don't want to resize /srv/ as that's already in use for zuul-merger (and is currently using > half the available disk space). Ideally we'd be able to add a disk to this machine just to store images.

See progress on T207707 . The new disks are mounted now and /mnt/docker can be used.

But per T207707#5304252 i would say we should NOT use devicemapper but stick to overlay2.

So i think one part of this ticket is duplicate of T207707 and the other part should be declined which would mean we can close this.

Indeed. What Daniel has said :-]