Page MenuHomePhabricator
Create Task
Maniphest T141126

Investigate moving docker to use direct-lvm devicemapper storage driver
Closed, ResolvedPublic
Actions

Description

Currently using the loopback mode driver, which isn't recommended for production systems https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/.

Details

SubjectRepoBranchLines +/-
k8s: Use direct-lvm for docker storage backendoperations/puppetproduction+51 -14
Customize query in gerrit

Related Objects

Event Timeline

yuvipanda created this task.Jul 22 2016, 4:56 PM
Restricted Application added a project: Cloud-Services. · View Herald TranscriptJul 22 2016, 4:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
yuvipanda mentioned this in T141017: Kubernetes worker nodes hanging.Jul 22 2016, 4:56 PM
• chasemp triaged this task as Medium priority.Jul 25 2016, 2:00 PM
yuvipanda added a comment.Jul 25 2016, 6:45 PM

I tried this and am at:

daemon: error initializing graphdriver: devmapper: Device docker-thinpool is not a thin pool"

I followed all the instructions in https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/.

I see there's a thinpool:

root@tools-docker-builder-01:/home/yuvipanda# lvdisplay 
  --- Logical volume ---
  LV Name                thinpool
  VG Name                docker
  LV UUID                1OAM1k-H1e5-eE2i-Qwsa-PD92-6ix3-2v12ZY
  LV Write Access        read/write
  LV Creation host, time tools-docker-builder-01, 2016-07-25 18:28:41 +0000
  LV Pool metadata       thinpool_tmeta
  LV Pool data           thinpool_tdata
  LV Status              available
  # open                 1
  LV Size                133.47 GiB
  Allocated pool data    0.00%
  Allocated metadata     0.01%
  Current LE             34168
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2
yuvipanda added a comment.Jul 28 2016, 11:52 PM

On further digging, here's what needs to happen:

  1. Delete /dev/vda4 which we create to be an lvm partition
  2. Recreate /dev/vda4 to be an unformatted partition
  3. *then* follow https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/

This makes it work :)

Now we need to write a script that automates this, and also probably figure out wether we should stick to xfs or switch to ext4

yuvipanda claimed this task.Jul 29 2016, 6:34 AM
yuvipanda raised the priority of this task from Medium to High.
yuvipanda added a comment.Jul 29 2016, 7:09 AM

Step 1:
Create a new instance

Step 2:

# parted /dev/vda unit B print
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 42949672960B
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start         End           Size          File system     Name     Flags
 1      17408B        1049087B      1031680B                      primary  bios_grub
 2      1049088B      536871423B    535822336B    linux-swap(v1)  swap
 3      536871424B    20937948671B  20401077248B  ext4            root
 4      20937965568B  42948624383B  22010658816B                  primary  lvm
root@tools-docker-test-02:/home/yuvipanda# vgdisplay
  --- Volume group ---
  VG Name               vd
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               20.50 GiB
  PE Size               4.00 MiB
  Total PE              5247
  Alloc PE / Size       0 / 0   
  Free  PE / Size       5247 / 20.50 GiB
  VG UUID               vOcWET-Gqhg-rl7I-pNZh-NCST-rxww-grxDha
root@tools-docker-test-02:/home/yuvipanda# lvdisplay
root@tools-docker-test-02:/home/yuvipanda#
root@tools-docker-test-02:/home/yuvipanda# pvdisplay
  --- Physical volume ---
  PV Name               /dev/vda4
  VG Name               vd
  PV Size               20.50 GiB / not usable 3.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              5247
  Free PE               5247
  Allocated PE          0
  PV UUID               1zhUuN-mBh4-mfaP-XkvI-3uHz-NKLk-i2NNDj

Step 3:

Cleanup the current VGs.

  1. vgremove vd
  2. pvremove /dev/vda4
  3. vgadisplay -> No volume groups found
  4. pvdisplay -> Empty

Step 4:
Follow https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/

  1. pvcreate /dev/vda4
  2. vgcreate docker /dev/vda4
  3. lvcreate --wipesignatures y -n thinpool docker -l 95%VG
  4. lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG
  5. lvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta
  6. (Skip the autogrow / monitor stuff for now, unnecessary)

This creates a thin pool:

root@tools-docker-test-02:/home/yuvipanda# vgdisplay
  --- Volume group ---
  VG Name               docker
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               20.50 GiB
  PE Size               4.00 MiB
  Total PE              5247
  Alloc PE / Size       5088 / 19.88 GiB
  Free  PE / Size       159 / 636.00 MiB
  VG UUID               kfIVWq-8LWe-JvZv-FZHd-U6zl-n2Rk-kdhvbO
root@tools-docker-test-02:/home/yuvipanda# lvs
  LV       VG     Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  thinpool docker twi-a-t--- 19.47g             0.00   0.03
root@tools-docker-test-02:/home/yuvipanda# pvdisplay
  --- Physical volume ---
  PV Name               /dev/vda4
  VG Name               docker
  PV Size               20.50 GiB / not usable 3.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              5247
  Free PE               159
  Allocated PE          5088
  PV UUID               wZ55bD-aYEx-sLP2-XaPR-kCSQ-Bnt3-2n1Lvp
root@tools-docker-test-02:/home/yuvipanda# parted /dev/vda unit B print
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 42949672960B
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start         End           Size          File system     Name     Flags
 1      17408B        1049087B      1031680B                      primary  bios_grub
 2      1049088B      536871423B    535822336B    linux-swap(v1)  swap
 3      536871424B    20937948671B  20401077248B  ext4            root
 4      20937965568B  42948624383B  22010658816B                  primary  lvm

Starting docker with the appropriate options:

--storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt dm.use_deferred_removal=true

results in failure:

Error starting daemon: error initializing graphdriver: devmapper: Device docker-thinpool is not a thin pool

Step 5:

DESTROY ALL THE THINGS

  1. vgremove docker
  2. pvremove /dev/vda4
  3. parted /dev/vda rm 4
  4. parted /dev/vda unit B print
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 42949672960B
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start       End           Size          File system     Name     Flags
 1      17408B      1049087B      1031680B                      primary  bios_grub
 2      1049088B    536871423B    535822336B    linux-swap(v1)  swap
 3      536871424B  20937948671B  20401077248B  ext4            root

Note we only have 3 partitions now!

Step 6:

CREATE FROM SCRATCH

  1. parted -s /dev/vda mkpart primary 20937948672B 100% (the 20937948672B is the end of the last partition from above +1)
  2. parted /dev/vda unit B print
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 42949672960B
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start         End           Size          File system     Name     Flags
 1      17408B        1049087B      1031680B                      primary  bios_grub
 2      1049088B      536871423B    535822336B    linux-swap(v1)  swap
 3      536871424B    20937948671B  20401077248B  ext4            root
 4      20937948672B  42949656063B  22011707392B                  primary

Step 7:

Setup docker thin volume again.

  1. pvcreate /dev/vda4
  2. vgcreate docker /dev/vda4
  3. lvcreate --wipesignatures y -n thinpool docker -l 95%VG
  4. lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG
  5. lvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta
root@tools-docker-test-02:/home/yuvipanda# lvs
  LV       VG     Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  thinpool docker twi-a-t--- 19.47g             0.00   0.03
root@tools-docker-test-02:/home/yuvipanda# vgdisplay
  --- Volume group ---
  VG Name               docker
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  5
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               20.50 GiB
  PE Size               4.00 MiB
  Total PE              5247
  Alloc PE / Size       5088 / 19.88 GiB
  Free  PE / Size       159 / 636.00 MiB
  VG UUID               ZkzHFc-43Jf-sx02-jYOy-5ZLQ-zs2S-jEk3SB

root@tools-docker-test-02:/home/yuvipanda# pvdisplay

  • Physical volume --- PV Name /dev/vda4 VG Name docker PV Size 20.50 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 5247 Free PE 159 Allocated PE 5088 PV UUID cL463K-g7iB-Kwqt-7B0C-nhwj-wPuW-GrH6Y8
So the last time I tried something akin to this I got docker to work, but this time it didn't - lol? So I'm going to try again tomorrow. Useful debug steps anyway.
yuvipanda added a comment.Jul 29 2016, 7:43 AM

That was useful, since that meant that the parted stuff was a red herring. I've a shorter way to get this to work just now:

  1. vgremove vd
  2. vgcreate direct-lvm /dev/vda4
  3. lvcreate --wipesignatures y -n data direct-lvm -l 95%VG
  4. lvcreate --wipesignatures y -n metadata direct-lvm -l 5%VG
  5. service docker stop
  6. --storage-opt dm.datadev=/dev/direct-lvm/data --storage-opt dm.metadatadev=/dev/direct-lvm/metadata params to docker daemon
  7. service docker start

This still errors, but a reboot fixes it! The difference from the previous thing seems to be primarily that we're specifying datadev and metadatadev separately, and not setting up thin volume ourselves...

yuvipanda added a comment.Jul 29 2016, 8:04 AM

No restart solution!

  1. lvcreate --wipesignatures y -n data vd-l 95%VG
  2. lvcreate --wipesignatures y -n metadata direct-lvm -l 5%VG
  3. service docker stop
  4. rm -rf /var/lib/docker/*
  5. Add --storage-opt dm.datadev=/dev/vd/data --storage-opt dm.metadatadev=/dev/vd/metadata to systemd params
  6. Start docker!

So the basica underlying issue seems to be:

  1. Make sure docker daemon is stopped during this
  2. Make sure that none of /var/lib/docker that was ever used with loopback driver survive for direct-lvm.
gerritbot subscribed.Jul 29 2016, 5:54 PM

Change 301853 had a related patch set uploaded (by Yuvipanda):
k8s: Use direct-lvm for docker storage backend

https://gerrit.wikimedia.org/r/301853

yuvipanda mentioned this in rOPUP945951b967ae: k8s: Use direct-lvm for docker storage backend.Jul 29 2016, 5:59 PM
yuvipanda mentioned this in rOPUP3f9bf3853ab4: k8s: Use direct-lvm for docker storage backend.
yuvipanda mentioned this in rOPUP3744d6bca5c1: k8s: Use direct-lvm for docker storage backend.Jul 29 2016, 6:04 PM
yuvipanda mentioned this in rOPUP2f6aee619061: k8s: Use direct-lvm for docker storage backend.
yuvipanda mentioned this in rOPUP306a849aae23: k8s: Use direct-lvm for docker storage backend.Jul 29 2016, 6:09 PM
yuvipanda mentioned this in rOPUPe4cd5ddec6b0: k8s: Use direct-lvm for docker storage backend.Jul 29 2016, 6:14 PM
yuvipanda mentioned this in rOPUP3ac96886421e: k8s: Use direct-lvm for docker storage backend.Jul 29 2016, 7:09 PM
yuvipanda mentioned this in rOPUP9fb702f2155c: k8s: Use direct-lvm for docker storage backend.Jul 29 2016, 8:09 PM
yuvipanda mentioned this in rOPUP5eb129a21e0e: k8s: Use direct-lvm for docker storage backend.Jul 29 2016, 8:14 PM
yuvipanda mentioned this in rOPUP628614d2966d: k8s: Use direct-lvm for docker storage backend.Jul 29 2016, 8:24 PM
yuvipanda mentioned this in rOPUPf8b3d4cc90e4: k8s: Use direct-lvm for docker storage backend.
yuvipanda mentioned this in rOPUPfad6433743a2: k8s: Use direct-lvm for docker storage backend.Jul 29 2016, 8:44 PM
gerritbot added a comment.Jul 29 2016, 8:54 PM

Change 301853 merged by Yuvipanda:
k8s: Use direct-lvm for docker storage backend

https://gerrit.wikimedia.org/r/301853

yuvipanda mentioned this in rOPUPd69cf9fa230d: k8s: Use direct-lvm for docker storage backend.Jul 29 2016, 8:59 PM
Stashbot subscribed.Aug 1 2016, 10:48 PM

Mentioned in SAL [2016-08-01T22:48:34Z] <madhuvishy> Depooling tools-worker-1009 to prepare for T141126

Stashbot added a comment.Aug 2 2016, 12:01 AM

Mentioned in SAL [2016-08-02T00:01:00Z] <yuvipanda> depool tools-worker-1017 for T141126

Stashbot added a comment.Aug 2 2016, 10:41 PM

Mentioned in SAL [2016-08-02T22:41:30Z] <madhuvishy> Depooling tools-worker 1012 and 1013 for T141126

Stashbot added a comment.Aug 2 2016, 10:44 PM

Mentioned in SAL [2016-08-02T22:44:43Z] <yuvipanda> depool tools-worker-1015 for T141126

Stashbot added a comment.Aug 2 2016, 10:49 PM

Mentioned in SAL [2016-08-02T22:49:32Z] <yuvipanda> depooled tools-worker-1014 as well for T141126

Stashbot added a comment.Aug 3 2016, 12:20 AM

Mentioned in SAL [2016-08-03T00:20:46Z] <madhuvishy> Repooled nodes tools-worker 1012 and 1013 for T141126

yuvipanda closed this task as Resolved.Aug 4 2016, 7:10 PM

I'm going to call this done because there are no more worker nodes left with the loopback configuration.

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits