Page MenuHomePhabricator

Switch CI Docker Storage Driver to its own partition and to use devicemapper
Open, LowPublic

Description

On contint1001 Docker uses the overlay engine on the root partition (/var/lib/docker/overlay2). That should be moved to a dedicated LVS logical volume and the backend switched to devicemapper.

from docker info:

Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true

If I remember properly, we should switch to devicemapper.

We have installed Docker via the upstream Debian package 17.06.2~ce-0~debian. It seems to select overlay2 by default.

Event Timeline

hashar created this task.Oct 20 2017, 9:22 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 20 2017, 9:22 AM

Looks like profile::docker::storage has the logic to setup a partition with parameters:

# list of physical volumes to use.
$physical_volumes = hiera('profile::docker::storage::physical_volumes'),
# Volume group to substitute.
$vg_to_remove = hiera('profile::docker::storage::vg_to_remove'),

It seems to create a new volume group docker with logical volumes data and metadata.

profile::docker::storage::physical_volumes would be the physical volume.

contint1001 has a 1TB disk and all the physical volume / volume group is allocated:

# pvdisplay
  --- Physical volume ---
  PV Name               /dev/md2
  VG Name               contint1001-vg
  PV Size               883.89 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
# vgdisplay 
  --- Volume group ---
  VG Name               contint1001-vg
  Format                lvm2
  VG Status             resizable
  Cur LV                1
  VG Size               883.89 GiB
root@contint1001:~# lvdisplay 
  --- Logical volume ---
  LV Path                /dev/contint1001-vg/data
  LV Name                data
  VG Name                contint1001-vg
  LV Size                883.89 GiB

Seems to me we would have to shrink the logical volume /dev/contint1001-vg/data and the volume group contint1001-vg.

hashar triaged this task as Low priority.Oct 27 2017, 8:40 PM

low priority since right now overlay2 does not seem to be an issue.

dduvall moved this task from Backlog to CI on the Release Pipeline board.Nov 6 2017, 6:18 PM
hashar renamed this task from Switch CI Docker Storage Driver to devicemapper to Switch CI Docker Storage Driver to its own partition and to use devicemapper.Mar 21 2018, 8:07 AM
hashar updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2018-03-21T08:09:34Z] <hashar> contint1001: docker image prune ; docker container prune # T178663

Dzahn added a subscriber: Dzahn.Jun 1 2018, 8:50 PM

contint1001 was close to running out of disk space today, Icinga alerted us.

Turns out it was due to docker images being rebuilt and it uses /var/lib/docker on / which has 46G of which 96% were in use.

At the same time contint1001 has a separate mount on /srv with over 400G free that is used only 51%.

In puppet the /var/lib/docker path appears in:

modules/profile/manifests/docker/storage/loopback.pp:# [*dm_source*] Source device for the /var/lib/docker directory
modules/profile/manifests/docker/storage/loopback.pp: $dm_target = '/var/lib/docker'

Can /srv/ be used instead for this?

hashar added a comment.Jun 1 2018, 8:54 PM

From my previous comment T178663#3699074 , I would love Docker material to be on a standalone partition. Namely shrink /srv somehow and create a new one. Then we can probably change the Docker storage driver to devicemapper and docker pull all the images we need from the registry.

Added T207707 as a subtask since it is about getting additional storage space without specifying any movement of the storage driver.

We don't want to resize /srv/ as that's already in use for zuul-merger (and is currently using > half the available disk space). Ideally we'd be able to add a disk to this machine just to store images.