Page MenuHomePhabricator

Udev (?) problems using modified swift puppet classes on deployment-ms-be0[56]
Closed, ResolvedPublic

Description

These instances are running stretch and I intend to replace deployment-ms-be0[34] which run jessie.
However, I seem to have run into a problem involving udev and the long-standing cherry-pick of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/361648/:

1krenair@deployment-ms-be05:~$ cat /etc/udev/rules.d/40-swift_storage_labs.rules
2ENV{DM_LV_NAME}=="lv-a1", ENV{DM_VG_NAME}=="vd", SYMLINK+="swift/lv-a1"krenair@deployment-ms-be05:~$
3krenair@deployment-ms-be05:~$ udevadm info /dev/dm-0 | grep DM_LV_NAME
4E: DM_LV_NAME=lv-a1
5krenair@deployment-ms-be05:~$ udevadm info /dev/dm-0 | grep DM_VG_NAME
6E: DM_VG_NAME=vd
7krenair@deployment-ms-be05:~$ udevadm info /dev/dm-0 | grep S: | grep swift
8krenair@deployment-ms-be05:~$ ls /dev/swift/ -lh
9ls: cannot access '/dev/swift/': No such file or directory

which of course causes:

Notice: /Stage[main]/Role::Swift::Storage/Swift::Init_device[lv-a]/Exec[parted-/dev/swift/lv-a1]/returns: Error: Could not stat device /dev/swift/lv-a - No such file or directory.
Notice: /Stage[main]/Role::Swift::Storage/Swift::Init_device[lv-a]/Exec[mkfs-/dev/swift/lv-a1]: Dependency Exec[parted-/dev/swift/lv-a1] has failures: true
Notice: /Stage[main]/Role::Swift::Storage/Swift::Init_device[lv-a]/Swift::Mount_filesystem[lv-a1]/File[mountpoint-/srv/swift-storage/lv-a1]: Dependency Exec[parted-/dev/swift/lv-a1] has failures: true
Error: parted --script --align optimal /dev/swift/lv-a mklabel gpt mkpart swift-lv-a1 1M 100% returned 1 instead of one of [0]
Error: /Stage[main]/Role::Swift::Storage/Swift::Init_device[lv-a]/Exec[parted-/dev/swift/lv-a1]: Could not evaluate: Puppet::Util::Log requires a message

Event Timeline

Krenair created this task.Apr 14 2019, 12:01 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 14 2019, 12:01 AM

As far as I can tell the puppet patch is fine, I can't explain udev not making that symlink but maybe I just don't know enough about udev.

Krenair renamed this task from Problems using modified swift puppet classes on deployment-ms-be0[56] to Udev (?) problems using modified swift puppet classes on deployment-ms-be0[56].Apr 14 2019, 12:07 AM

Interestingly sudo udevadm test /devices/virtual/block/dm-0 on 03 says LINK 'swift/lv-a1' /etc/udev/rules.d/90-swift-lvm.rules:1 which is a seemingly unpuppetised file virtually identical to /etc/udev/rules.d/40-swift_storage_labs.rules, just with a trailing newline. It thinks /etc/udev/rules.d/40-swift_storage_labs.rules is invalid due to the lack of trailing newline, but when I fix that it still picks the unpuppetised file.
If I comment the unpuppetised file it deletes the symlink, but they're identical now so... Maybe our rules only work on the basis of some rule in a file with priority between 40 and 90?

I bet we need a priority > 56 for our rule using DM_LV_NAME/DM_VG_NAME:

krenair@deployment-ms-be03:~$ grep DM_LV_NAME /lib/udev/rules.d/*
/lib/udev/rules.d/56-lvm.rules:#   DM_LV_NAME - logical volume name
/lib/udev/rules.d/56-lvm.rules:ENV{DM_LV_NAME}=="pvmove?*|?*_vorigin", GOTO="lvm_disable"
/lib/udev/rules.d/56-lvm.rules:ENV{DM_VG_NAME}=="?*", ENV{DM_LV_NAME}=="?*", SYMLINK+="$env{DM_VG_NAME}/$env{DM_LV_NAME}", GOTO="lvm_end"
krenair@deployment-ms-be03:~$ grep DM_VG_NAME /lib/udev/rules.d/*
/lib/udev/rules.d/56-lvm.rules:#   DM_VG_NAME - volume group name
/lib/udev/rules.d/56-lvm.rules:ENV{DM_VG_NAME}=="?*", ENV{DM_LV_NAME}=="?*", SYMLINK+="$env{DM_VG_NAME}/$env{DM_LV_NAME}", GOTO="lvm_end"

04 also has another dupe of this with priority 99

Change 503707 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Fix broken profile::swift::storage::labs

https://gerrit.wikimedia.org/r/503707

Anyway with all of that out of the way, udev works, puppet works, and after a reboot there is /dev/mapper/vd-lv--a1 113G 33M 113G 1% /srv/swift-storage/lv-a1

Change 503707 merged by Filippo Giunchedi:
[operations/puppet@production] Fix broken profile::swift::storage::labs

https://gerrit.wikimedia.org/r/503707

Krenair closed this task as Resolved.Apr 18 2019, 1:40 PM