Page MenuHomePhabricator
Create Task
Maniphest T220895

Udev (?) problems using modified swift puppet classes on deployment-ms-be0[56]
Closed, ResolvedPublic
Actions

Description

These instances are running stretch and I intend to replace deployment-ms-be0[34] which run jessie.
However, I seem to have run into a problem involving udev and the long-standing cherry-pick of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/361648/:

P8393 labs swift udev problems
1krenair@deployment-ms-be05:~$ cat /etc/udev/rules.d/40-swift_storage_labs.rules
2ENV{DM_LV_NAME}=="lv-a1", ENV{DM_VG_NAME}=="vd", SYMLINK+="swift/lv-a1"krenair@deployment-ms-be05:~$
3krenair@deployment-ms-be05:~$ udevadm info /dev/dm-0 | grep DM_LV_NAME
4E: DM_LV_NAME=lv-a1
5krenair@deployment-ms-be05:~$ udevadm info /dev/dm-0 | grep DM_VG_NAME
6E: DM_VG_NAME=vd
7krenair@deployment-ms-be05:~$ udevadm info /dev/dm-0 | grep S: | grep swift
8krenair@deployment-ms-be05:~$ ls /dev/swift/ -lh
9ls: cannot access '/dev/swift/': No such file or directory

which of course causes:

Notice: /Stage[main]/Role::Swift::Storage/Swift::Init_device[lv-a]/Exec[parted-/dev/swift/lv-a1]/returns: Error: Could not stat device /dev/swift/lv-a - No such file or directory.
Notice: /Stage[main]/Role::Swift::Storage/Swift::Init_device[lv-a]/Exec[mkfs-/dev/swift/lv-a1]: Dependency Exec[parted-/dev/swift/lv-a1] has failures: true
Notice: /Stage[main]/Role::Swift::Storage/Swift::Init_device[lv-a]/Swift::Mount_filesystem[lv-a1]/File[mountpoint-/srv/swift-storage/lv-a1]: Dependency Exec[parted-/dev/swift/lv-a1] has failures: true
Error: parted --script --align optimal /dev/swift/lv-a mklabel gpt mkpart swift-lv-a1 1M 100% returned 1 instead of one of [0]
Error: /Stage[main]/Role::Swift::Storage/Swift::Init_device[lv-a]/Exec[parted-/dev/swift/lv-a1]: Could not evaluate: Puppet::Util::Log requires a message

Details

SubjectRepoBranchLines +/-
Fix broken profile::swift::storage::labsoperations/puppetproduction+3 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

Krenair created this task.Apr 14 2019, 12:01 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 14 2019, 12:01 AM
Krenair added a comment.Apr 14 2019, 12:05 AM

As far as I can tell the puppet patch is fine, I can't explain udev not making that symlink but maybe I just don't know enough about udev.

Krenair renamed this task from Problems using modified swift puppet classes on deployment-ms-be0[56] to Udev (?) problems using modified swift puppet classes on deployment-ms-be0[56].Apr 14 2019, 12:07 AM
Krenair added a comment.Apr 14 2019, 1:31 AM

Interestingly sudo udevadm test /devices/virtual/block/dm-0 on 03 says LINK 'swift/lv-a1' /etc/udev/rules.d/90-swift-lvm.rules:1 which is a seemingly unpuppetised file virtually identical to /etc/udev/rules.d/40-swift_storage_labs.rules, just with a trailing newline. It thinks /etc/udev/rules.d/40-swift_storage_labs.rules is invalid due to the lack of trailing newline, but when I fix that it still picks the unpuppetised file.
If I comment the unpuppetised file it deletes the symlink, but they're identical now so... Maybe our rules only work on the basis of some rule in a file with priority between 40 and 90?

Krenair added a comment.Apr 14 2019, 1:34 AM

I bet we need a priority > 56 for our rule using DM_LV_NAME/DM_VG_NAME:

krenair@deployment-ms-be03:~$ grep DM_LV_NAME /lib/udev/rules.d/*
/lib/udev/rules.d/56-lvm.rules:#   DM_LV_NAME - logical volume name
/lib/udev/rules.d/56-lvm.rules:ENV{DM_LV_NAME}=="pvmove?*|?*_vorigin", GOTO="lvm_disable"
/lib/udev/rules.d/56-lvm.rules:ENV{DM_VG_NAME}=="?*", ENV{DM_LV_NAME}=="?*", SYMLINK+="$env{DM_VG_NAME}/$env{DM_LV_NAME}", GOTO="lvm_end"
krenair@deployment-ms-be03:~$ grep DM_VG_NAME /lib/udev/rules.d/*
/lib/udev/rules.d/56-lvm.rules:#   DM_VG_NAME - volume group name
/lib/udev/rules.d/56-lvm.rules:ENV{DM_VG_NAME}=="?*", ENV{DM_LV_NAME}=="?*", SYMLINK+="$env{DM_VG_NAME}/$env{DM_LV_NAME}", GOTO="lvm_end"
Krenair claimed this task.Apr 14 2019, 1:38 AM

yeah

Krenair added a comment.Apr 14 2019, 1:39 AM

04 also has another dupe of this with priority 99

gerritbot added a comment.Apr 14 2019, 1:45 AM

Change 503707 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Fix broken profile::swift::storage::labs

https://gerrit.wikimedia.org/r/503707

gerritbot added a project: Patch-For-Review.Apr 14 2019, 1:45 AM
Krenair added a comment.Apr 14 2019, 2:05 AM

I just found https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/371642/5/modules/profile/manifests/swift/storage/labs.pp - it would've worked fine until that commit

Krenair added a comment.Apr 14 2019, 2:23 AM

Anyway with all of that out of the way, udev works, puppet works, and after a reboot there is /dev/mapper/vd-lv--a1 113G 33M 113G 1% /srv/swift-storage/lv-a1

gerritbot added a comment.Apr 18 2019, 10:04 AM

Change 503707 merged by Filippo Giunchedi:
[operations/puppet@production] Fix broken profile::swift::storage::labs

https://gerrit.wikimedia.org/r/503707

Krenair closed this task as Resolved.Apr 18 2019, 1:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL