unstable device mapping of SSDs causing swift/puppet problems - example reimage
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	MatthewVernon
	May 18 2022, 9:15 AM

Description

We talked about this the other day, and I was asked to produce an example of how the reimage process works. In this case, the installer worked first time, so no filesystems were destroyed, but it took multiple reboots before puppet could run OK.

Host is ms-be2056. State before reimage:

mvernon@ms-be2056:~$ df -lh
Filesystem      Size  Used Avail Use% Mounted on
udev             63G     0   63G   0% /dev
tmpfs            13G  1.4G   12G  11% /run
/dev/md0         55G  6.3G   46G  12% /
tmpfs            63G  4.0K   63G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            63G     0   63G   0% /sys/fs/cgroup
/dev/sdb3        94G   53G   41G  57% /srv/swift-storage/sdb3
/dev/sdj1       3.7T  2.7T  1.1T  73% /srv/swift-storage/sdj1
/dev/sdi1       3.7T  2.6T  1.1T  72% /srv/swift-storage/sdi1
/dev/sdb4       298G  340M  297G   1% /srv/swift-storage/sdb4
/dev/sdk1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdk1
/dev/sdh1       3.7T  2.7T 1012G  73% /srv/swift-storage/sdh1
/dev/sda4       298G  420M  297G   1% /srv/swift-storage/sda4
/dev/sda3        94G   33G   61G  36% /srv/swift-storage/sda3
/dev/sdg1       3.7T  2.6T  1.1T  72% /srv/swift-storage/sdg1
/dev/sdm1       3.7T  2.7T  1.1T  73% /srv/swift-storage/sdm1
/dev/sde1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sde1
/dev/sdn1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdn1
/dev/sdf1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdf1
/dev/sdd1       3.7T  2.7T  1.1T  73% /srv/swift-storage/sdd1
/dev/sdc1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdc1
/dev/sdl1       3.7T  2.6T  1.1T  72% /srv/swift-storage/sdl1
tmpfs            13G     0   13G   0% /run/user/33349
mvernon@ms-be2056:~$ cat /proc/mdstat 
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sda1[0] sdb1[1]
      58559488 blocks super 1.2 [2/2] [UU]
      
md1 : active raid1 sda2[0] sdb2[1]
      976320 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>
mvernon@ms-be2056:~$ cat /sys/block/sd{a,b}/queue/rotational 
0
0
mvernon@ms-be2056:~$ sudo blkid
/dev/md0: UUID="a524364d-5b7f-42ab-9a02-ff3c4bdd77b1" TYPE="ext4"
/dev/sdb1: UUID="f1e97e51-9e01-201e-c003-378d5955f0ee" UUID_SUB="6e6ee8af-d5ea-9878-fb4e-c1e5a40f61b4" LABEL="ms-be2056:0" TYPE="linux_raid_member" PARTUUID="c7778797-01"
/dev/sdb2: UUID="461b3ee6-6965-21f7-a5c9-d3dbf18cd1fd" UUID_SUB="d7f88edc-8052-3c7e-8a07-492cbb88765e" LABEL="ms-be2056:1" TYPE="linux_raid_member" PARTUUID="c7778797-02"
/dev/sdb3: LABEL="swift-sdb3" UUID="7b3e0116-ad4c-4050-9b9f-99a4573eb6b2" TYPE="xfs" PARTUUID="c7778797-03"
/dev/sdb4: LABEL="swift-sdb4" UUID="949e4884-2d23-4bde-be3f-b7ccc0cb4d96" TYPE="xfs" PARTUUID="c7778797-04"
/dev/sdc1: LABEL="swift-sdc1" UUID="37516634-0e19-47ed-8e10-1eac7145019f" TYPE="xfs" PARTLABEL="swift-sdc1" PARTUUID="c13f31af-6282-4929-9c6e-1d85369edbd2"
/dev/sde1: LABEL="swift-sde1" UUID="c8309cd7-3656-47b7-866e-df4d2b31db43" TYPE="xfs" PARTLABEL="swift-sde1" PARTUUID="09e14ff5-76f6-44f5-bd4f-c440c1a8dcc4"
/dev/sdd1: LABEL="swift-sdd1" UUID="a00571e1-46a6-435b-a943-476519c465f5" TYPE="xfs" PARTLABEL="swift-sdd1" PARTUUID="6d9ef4c1-f5ba-4e37-9799-808fa0c79d6f"
/dev/sdf1: LABEL="swift-sdf1" UUID="de078d5c-a82a-45a0-9132-7d592a57a9ae" TYPE="xfs" PARTLABEL="swift-sdf1" PARTUUID="fd896ff7-6ca0-4189-83f9-74a580c9965a"
/dev/sdg1: LABEL="swift-sdg1" UUID="122f9d94-470f-4d15-96b9-085d660439ed" TYPE="xfs" PARTLABEL="swift-sdg1" PARTUUID="3e37ce6a-603a-43bf-8b35-acb349d86278"
/dev/sdh1: LABEL="swift-sdh1" UUID="c7ada158-d982-4b0a-b702-fe97c5bd199f" TYPE="xfs" PARTLABEL="swift-sdh1" PARTUUID="8c0ca9f1-d24b-4872-8d7a-b61df14b4055"
/dev/sdi1: LABEL="swift-sdi1" UUID="145acd7d-d3a2-48de-8129-57440aa083a9" TYPE="xfs" PARTLABEL="swift-sdi1" PARTUUID="f552416a-8ccc-40cf-abd5-76182acbd73f"
/dev/sdj1: LABEL="swift-sdj1" UUID="253f649f-32d3-4e15-b3e9-4bdc678af280" TYPE="xfs" PARTLABEL="swift-sdj1" PARTUUID="edfe319a-ed0b-442e-a882-01994c9af981"
/dev/sdk1: LABEL="swift-sdk1" UUID="9a139769-e1f2-414d-8a2f-6ef7c6fd15cc" TYPE="xfs" PARTLABEL="swift-sdk1" PARTUUID="2ceeca24-54a1-429e-b0e1-e67259c33da2"
/dev/sdl1: LABEL="swift-sdl1" UUID="9ed4006e-df64-416e-93c8-f5cfe69f2a33" TYPE="xfs" PARTLABEL="swift-sdl1" PARTUUID="98699112-e45c-410e-b6ed-c6901165a837"
/dev/sdm1: LABEL="swift-sdm1" UUID="d62cc7ec-c544-4c57-9d60-691bc9004b5d" TYPE="xfs" PARTLABEL="swift-sdm1" PARTUUID="9c4cd09b-803b-41d8-9fad-cdeb702dd43f"
/dev/sdn1: LABEL="swift-sdn1" UUID="a2ed23e7-8da0-4bfd-829d-cd2626228b50" TYPE="xfs" PARTLABEL="swift-sdn1" PARTUUID="6f4b131c-99d5-4624-b403-66a0467ca6d3"
/dev/md1: UUID="e98ce176-7140-42c1-91ac-973001bed936" TYPE="swap"
/dev/sda1: UUID="f1e97e51-9e01-201e-c003-378d5955f0ee" UUID_SUB="ee8eac20-a7aa-c8e1-9d1a-424e4197a3bd" LABEL="ms-be2056:0" TYPE="linux_raid_member" PARTUUID="d22f78f1-01"
/dev/sda2: UUID="461b3ee6-6965-21f7-a5c9-d3dbf18cd1fd" UUID_SUB="4dba7312-af1c-973f-25bd-c14c4c130568" LABEL="ms-be2056:1" TYPE="linux_raid_member" PARTUUID="d22f78f1-02"
/dev/sda3: LABEL="swift-sda3" UUID="08b55dd6-b7a9-48e7-8faa-26a74094ac3e" TYPE="xfs" PARTUUID="d22f78f1-03"
/dev/sda4: LABEL="swift-sda4" UUID="878042ed-8ac5-4a5c-a884-545b51821382" TYPE="xfs" PARTUUID="d22f78f1-04"

Install worked OK, but first post-install boot, ssds have appeared as sdc and sda, so puppet doesn't work:

mvernon@ms-be2056:~$ df -lh
Filesystem      Size  Used Avail Use% Mounted on
udev             63G     0   63G   0% /dev
tmpfs            13G  1.8M   13G   1% /run
/dev/md0         55G  2.5G   50G   5% /
tmpfs            63G  4.0K   63G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sdc3        94G   51G   43G  55% /srv/swift-storage/sdb3
/dev/sdc4       298G  2.2G  295G   1% /srv/swift-storage/sdb4
/dev/sda3        94G   25G   69G  27% /srv/swift-storage/sda3
/dev/sda4       298G  2.2G  295G   1% /srv/swift-storage/sda4
/dev/sdd1       3.7T  2.7T  1.1T  73% /srv/swift-storage/sdd1
/dev/sde1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdf1
/dev/sdl1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdk1
/dev/sdh1       3.7T  2.6T  1.1T  72% /srv/swift-storage/sdi1
/dev/sdi1       3.7T  2.7T 1012G  73% /srv/swift-storage/sdh1
/dev/sdb1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdc1
/dev/sdg1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sde1
/dev/sdk1       3.7T  2.7T  1.1T  73% /srv/swift-storage/sdj1
/dev/sdj1       3.7T  2.6T  1.1T  72% /srv/swift-storage/sdl1
/dev/sdf1       3.7T  2.6T  1.1T  72% /srv/swift-storage/sdg1
/dev/sdm1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdn1
/dev/sdn1       3.7T  2.7T  1.1T  73% /srv/swift-storage/sdm1
tmpfs            13G     0   13G   0% /run/user/33349
mvernon@ms-be2056:~$ sudo blkid
/dev/sdc1: UUID="90122c2e-ad6c-c49b-95c8-137cada4013f" UUID_SUB="abc61a09-4578-6a96-3312-a9944ad609de" LABEL="ms-be2056:0" TYPE="linux_raid_member" PARTUUID="0cdc45e1-01"
/dev/sdc2: UUID="50d4e233-8d1b-74b5-91a0-9bd491f987d2" UUID_SUB="b57661c5-c157-6958-7728-af9a4e69ae33" LABEL="ms-be2056:1" TYPE="linux_raid_member" PARTUUID="0cdc45e1-02"
/dev/sdc3: LABEL="swift-sdb3" UUID="f2bca9d7-f896-4dec-834b-fea69f0c4cf1" BLOCK_SIZE="4096" TYPE="xfs" PARTUUID="0cdc45e1-03"
/dev/sdc4: LABEL="swift-sdb4" UUID="7bc81d15-2135-4d09-a23d-f03f28c7f055" BLOCK_SIZE="4096" TYPE="xfs" PARTUUID="0cdc45e1-04"
/dev/sda1: UUID="90122c2e-ad6c-c49b-95c8-137cada4013f" UUID_SUB="40812a83-a759-40af-8ea0-0e85cc562b8e" LABEL="ms-be2056:0" TYPE="linux_raid_member" PARTUUID="ff552554-01"
/dev/sda2: UUID="50d4e233-8d1b-74b5-91a0-9bd491f987d2" UUID_SUB="9a5ffdcb-0ddb-5db0-1e6d-85b6cfeacdcd" LABEL="ms-be2056:1" TYPE="linux_raid_member" PARTUUID="ff552554-02"
/dev/sda3: LABEL="swift-sda3" UUID="2ae842c7-d5ae-4ef7-8c52-214390d9958d" BLOCK_SIZE="4096" TYPE="xfs" PARTUUID="ff552554-03"
/dev/sda4: LABEL="swift-sda4" UUID="f5e754f7-341c-4f05-b23e-8900afb7fec4" BLOCK_SIZE="4096" TYPE="xfs" PARTUUID="ff552554-04"
/dev/sdf1: LABEL="swift-sdg1" UUID="122f9d94-470f-4d15-96b9-085d660439ed" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdg1" PARTUUID="3e37ce6a-603a-43bf-8b35-acb349d86278"
/dev/sdl1: LABEL="swift-sdk1" UUID="9a139769-e1f2-414d-8a2f-6ef7c6fd15cc" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdk1" PARTUUID="2ceeca24-54a1-429e-b0e1-e67259c33da2"
/dev/sdj1: LABEL="swift-sdl1" UUID="9ed4006e-df64-416e-93c8-f5cfe69f2a33" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdl1" PARTUUID="98699112-e45c-410e-b6ed-c6901165a837"
/dev/sdm1: LABEL="swift-sdn1" UUID="a2ed23e7-8da0-4bfd-829d-cd2626228b50" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdn1" PARTUUID="6f4b131c-99d5-4624-b403-66a0467ca6d3"
/dev/sde1: LABEL="swift-sdf1" UUID="de078d5c-a82a-45a0-9132-7d592a57a9ae" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdf1" PARTUUID="fd896ff7-6ca0-4189-83f9-74a580c9965a"
/dev/sdi1: LABEL="swift-sdh1" UUID="c7ada158-d982-4b0a-b702-fe97c5bd199f" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdh1" PARTUUID="8c0ca9f1-d24b-4872-8d7a-b61df14b4055"
/dev/sdh1: LABEL="swift-sdi1" UUID="145acd7d-d3a2-48de-8129-57440aa083a9" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdi1" PARTUUID="f552416a-8ccc-40cf-abd5-76182acbd73f"
/dev/sdn1: LABEL="swift-sdm1" UUID="d62cc7ec-c544-4c57-9d60-691bc9004b5d" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdm1" PARTUUID="9c4cd09b-803b-41d8-9fad-cdeb702dd43f"
/dev/sdg1: LABEL="swift-sde1" UUID="c8309cd7-3656-47b7-866e-df4d2b31db43" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sde1" PARTUUID="09e14ff5-76f6-44f5-bd4f-c440c1a8dcc4"
/dev/sdk1: LABEL="swift-sdj1" UUID="253f649f-32d3-4e15-b3e9-4bdc678af280" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdj1" PARTUUID="edfe319a-ed0b-442e-a882-01994c9af981"
/dev/sdd1: LABEL="swift-sdd1" UUID="a00571e1-46a6-435b-a943-476519c465f5" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdd1" PARTUUID="6d9ef4c1-f5ba-4e37-9799-808fa0c79d6f"
/dev/sdb1: LABEL="swift-sdc1" UUID="37516634-0e19-47ed-8e10-1eac7145019f" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdc1" PARTUUID="c13f31af-6282-4929-9c6e-1d85369edbd2"
/dev/md0: UUID="4267410e-7a5f-467e-b07d-b59c0349aefc" BLOCK_SIZE="4096" TYPE="ext4"
/dev/md1: UUID="84f6a334-7dec-4db6-bf86-0877e0b9ae1d" TYPE="swap"
mvernon@ms-be2056:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for ms-be2056.codfw.wmnet
Info: Applying configuration version '(7b5aa93da2) Jbond - nrpe: add nrpe::plugin to only installs scripts to hosts with nrpe'
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Init_device[/dev/sdc]/Exec[mkfs-/dev/sdc1]/returns: mkfs.xfs: cannot open /dev/sdc1: Device or resource busy
Error: 'mkfs -t xfs -L swift-sdc1 -m crc=1 -m finobt=0 -i size=512 /dev/sdc1' returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Swift::Storage/Swift::Init_device[/dev/sdc]/Exec[mkfs-/dev/sdc1]/returns: change from 'notrun' to ['0'] failed: 'mkfs -t xfs -L swift-sdc1 -m crc=1 -m finobt=0 -i size=512 /dev/sdc1' returned 1 instead of one of [0] (corrective)
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb3]/Exec[xfs_label-/dev/sdb3]/returns: /dev/sdb3: No such file or directory
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb3]/Exec[xfs_label-/dev/sdb3]/returns: 
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb3]/Exec[xfs_label-/dev/sdb3]/returns: fatal error -- couldn't initialize XFS library
Error: 'xfs_admin -L swift-sdb3 /dev/sdb3' returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb3]/Exec[xfs_label-/dev/sdb3]/returns: change from 'notrun' to ['0'] failed: 'xfs_admin -L swift-sdb3 /dev/sdb3' returned 1 instead of one of [0] (corrective)
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb4]/Exec[xfs_label-/dev/sdb4]/returns: /dev/sdb4: No such file or directory
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb4]/Exec[xfs_label-/dev/sdb4]/returns: 
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb4]/Exec[xfs_label-/dev/sdb4]/returns: fatal error -- couldn't initialize XFS library
Error: 'xfs_admin -L swift-sdb4 /dev/sdb4' returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb4]/Exec[xfs_label-/dev/sdb4]/returns: change from 'notrun' to ['0'] failed: 'xfs_admin -L swift-sdb4 /dev/sdb4' returned 1 instead of one of [0] (corrective)
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Init_device[/dev/sdc]/Swift::Mount_filesystem[/dev/sdc1]/Exec[mountpoint-root-/srv/swift-storage/sdc1]: Dependency Exec[mkfs-/dev/sdc1] has failures: true
Warning: /Stage[main]/Profile::Swift::Storage/Swift::Init_device[/dev/sdc]/Swift::Mount_filesystem[/dev/sdc1]/Exec[mountpoint-root-/srv/swift-storage/sdc1]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Swift::Storage/Swift::Init_device[/dev/sdc]/Swift::Mount_filesystem[/dev/sdc1]/Exec[mountpoint-swift-/srv/swift-storage/sdc1]: Skipping because of failed dependencies
Warning: /Stage[main]/Profile::Swift::Storage/Swift::Init_device[/dev/sdc]/Swift::Mount_filesystem[/dev/sdc1]/Mount[/srv/swift-storage/sdc1]: Skipping because of failed dependencies
Notice: Applied catalog in 24.53 seconds

This pattern (of sda appearing as sdc, and sdb appearing at sda) repeated through 2 further reboots, then sda and sdb were swapped, which still breaks puppet:

mvernon@ms-be2056:~$ df -lh
Filesystem      Size  Used Avail Use% Mounted on
udev             63G     0   63G   0% /dev
tmpfs            13G  1.8M   13G   1% /run
/dev/md0         55G  2.7G   50G   6% /
tmpfs            63G  4.0K   63G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda4       298G  2.2G  295G   1% /srv/swift-storage/sdb4
/dev/sdb4       298G  2.2G  295G   1% /srv/swift-storage/sda4
/dev/sdb3        94G   34G   61G  36% /srv/swift-storage/sda3
/dev/sda3        94G   70G   24G  76% /srv/swift-storage/sdb3
/dev/sdd1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sde1
/dev/sdg1       3.7T  2.6T  1.1T  72% /srv/swift-storage/sdg1
/dev/sdh1       3.7T  2.7T 1012G  73% /srv/swift-storage/sdh1
/dev/sde1       3.7T  2.7T  1.1T  73% /srv/swift-storage/sdd1
/dev/sdf1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdf1
/dev/sdm1       3.7T  2.7T  1.1T  73% /srv/swift-storage/sdm1
/dev/sdi1       3.7T  2.6T  1.1T  72% /srv/swift-storage/sdi1
/dev/sdn1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdn1
/dev/sdj1       3.7T  2.7T  1.1T  73% /srv/swift-storage/sdj1
/dev/sdc1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdc1
/dev/sdk1       3.7T  2.6T  1.1T  72% /srv/swift-storage/sdl1
/dev/sdl1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdk1
tmpfs            13G     0   13G   0% /run/user/33349
mvernon@ms-be2056:~$ sudo run-puppet-agent
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for ms-be2056.codfw.wmnet
Info: Applying configuration version '(21ccc4617c) Muehlenhoff - jenkins: update path to war file'
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sda3]/Exec[xfs_label-/dev/sda3]/returns: xfs_admin: /dev/sda3 contains a mounted filesystem
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sda3]/Exec[xfs_label-/dev/sda3]/returns: 
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sda3]/Exec[xfs_label-/dev/sda3]/returns: fatal error -- couldn't initialize XFS library
Error: 'xfs_admin -L swift-sda3 /dev/sda3' returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sda3]/Exec[xfs_label-/dev/sda3]/returns: change from 'notrun' to ['0'] failed: 'xfs_admin -L swift-sda3 /dev/sda3' returned 1 instead of one of [0] (corrective)
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sda4]/Exec[xfs_label-/dev/sda4]/returns: xfs_admin: /dev/sda4 contains a mounted filesystem
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sda4]/Exec[xfs_label-/dev/sda4]/returns: 
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sda4]/Exec[xfs_label-/dev/sda4]/returns: fatal error -- couldn't initialize XFS library
Error: 'xfs_admin -L swift-sda4 /dev/sda4' returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sda4]/Exec[xfs_label-/dev/sda4]/returns: change from 'notrun' to ['0'] failed: 'xfs_admin -L swift-sda4 /dev/sda4' returned 1 instead of one of [0] (corrective)
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb3]/Exec[xfs_label-/dev/sdb3]/returns: xfs_admin: /dev/sdb3 contains a mounted filesystem
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb3]/Exec[xfs_label-/dev/sdb3]/returns: 
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb3]/Exec[xfs_label-/dev/sdb3]/returns: fatal error -- couldn't initialize XFS library
Error: 'xfs_admin -L swift-sdb3 /dev/sdb3' returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb3]/Exec[xfs_label-/dev/sdb3]/returns: change from 'notrun' to ['0'] failed: 'xfs_admin -L swift-sdb3 /dev/sdb3' returned 1 instead of one of [0] (corrective)
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb4]/Exec[xfs_label-/dev/sdb4]/returns: xfs_admin: /dev/sdb4 contains a mounted filesystem
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb4]/Exec[xfs_label-/dev/sdb4]/returns: 
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb4]/Exec[xfs_label-/dev/sdb4]/returns: fatal error -- couldn't initialize XFS library
Error: 'xfs_admin -L swift-sdb4 /dev/sdb4' returned 1 instead of one of [0]
Error: /Stage[main]/Profile::Swift::Storage/Swift::Label_filesystem[/dev/sdb4]/Exec[xfs_label-/dev/sdb4]/returns: change from 'notrun' to ['0'] failed: 'xfs_admin -L swift-sdb4 /dev/sdb4' returned 1 instead of one of [0] (corrective)
Notice: Applied catalog in 24.57 seconds

After 3 further reboots, the SSDs were in the correct places:

mvernon@ms-be2056:~$ df -lh
Filesystem      Size  Used Avail Use% Mounted on
udev             63G     0   63G   0% /dev
tmpfs            13G  1.8M   13G   1% /run
/dev/md0         55G  2.9G   50G   6% /
tmpfs            63G  4.0K   63G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sdb3        94G   70G   24G  76% /srv/swift-storage/sdb3
/dev/sda3        94G   34G   61G  36% /srv/swift-storage/sda3
/dev/sda4       298G  2.2G  295G   1% /srv/swift-storage/sda4
/dev/sdb4       298G  2.2G  295G   1% /srv/swift-storage/sdb4
/dev/sdg1       3.7T  2.6T  1.1T  72% /srv/swift-storage/sdg1
/dev/sde1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sde1
/dev/sdi1       3.7T  2.7T 1012G  73% /srv/swift-storage/sdh1
/dev/sdd1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdc1
/dev/sdj1       3.7T  2.6T  1.1T  72% /srv/swift-storage/sdl1
/dev/sdc1       3.7T  2.7T  1.1T  73% /srv/swift-storage/sdd1
/dev/sdn1       3.7T  2.7T  1.1T  73% /srv/swift-storage/sdm1
/dev/sdf1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdf1
/dev/sdk1       3.7T  2.6T  1.1T  72% /srv/swift-storage/sdi1
/dev/sdm1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdn1
/dev/sdh1       3.7T  2.7T  1.1T  73% /srv/swift-storage/sdj1
/dev/sdl1       3.7T  2.7T  1.1T  72% /srv/swift-storage/sdk1
tmpfs            13G     0   13G   0% /run/user/33349
mvernon@ms-be2056:~$ sudo blkid
/dev/sda1: UUID="90122c2e-ad6c-c49b-95c8-137cada4013f" UUID_SUB="40812a83-a759-40af-8ea0-0e85cc562b8e" LABEL="ms-be2056:0" TYPE="linux_raid_member" PARTUUID="ff552554-01"
/dev/sda2: UUID="50d4e233-8d1b-74b5-91a0-9bd491f987d2" UUID_SUB="9a5ffdcb-0ddb-5db0-1e6d-85b6cfeacdcd" LABEL="ms-be2056:1" TYPE="linux_raid_member" PARTUUID="ff552554-02"
/dev/sda3: LABEL="swift-sda3" UUID="2ae842c7-d5ae-4ef7-8c52-214390d9958d" BLOCK_SIZE="4096" TYPE="xfs" PARTUUID="ff552554-03"
/dev/sda4: LABEL="swift-sda4" UUID="f5e754f7-341c-4f05-b23e-8900afb7fec4" BLOCK_SIZE="4096" TYPE="xfs" PARTUUID="ff552554-04"
/dev/sde1: LABEL="swift-sde1" UUID="c8309cd7-3656-47b7-866e-df4d2b31db43" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sde1" PARTUUID="09e14ff5-76f6-44f5-bd4f-c440c1a8dcc4"
/dev/sdh1: LABEL="swift-sdj1" UUID="253f649f-32d3-4e15-b3e9-4bdc678af280" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdj1" PARTUUID="edfe319a-ed0b-442e-a882-01994c9af981"
/dev/sdi1: LABEL="swift-sdh1" UUID="c7ada158-d982-4b0a-b702-fe97c5bd199f" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdh1" PARTUUID="8c0ca9f1-d24b-4872-8d7a-b61df14b4055"
/dev/sdm1: LABEL="swift-sdn1" UUID="a2ed23e7-8da0-4bfd-829d-cd2626228b50" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdn1" PARTUUID="6f4b131c-99d5-4624-b403-66a0467ca6d3"
/dev/sdf1: LABEL="swift-sdf1" UUID="de078d5c-a82a-45a0-9132-7d592a57a9ae" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdf1" PARTUUID="fd896ff7-6ca0-4189-83f9-74a580c9965a"
/dev/sdg1: LABEL="swift-sdg1" UUID="122f9d94-470f-4d15-96b9-085d660439ed" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdg1" PARTUUID="3e37ce6a-603a-43bf-8b35-acb349d86278"
/dev/sdj1: LABEL="swift-sdl1" UUID="9ed4006e-df64-416e-93c8-f5cfe69f2a33" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdl1" PARTUUID="98699112-e45c-410e-b6ed-c6901165a837"
/dev/sdl1: LABEL="swift-sdk1" UUID="9a139769-e1f2-414d-8a2f-6ef7c6fd15cc" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdk1" PARTUUID="2ceeca24-54a1-429e-b0e1-e67259c33da2"
/dev/sdn1: LABEL="swift-sdm1" UUID="d62cc7ec-c544-4c57-9d60-691bc9004b5d" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdm1" PARTUUID="9c4cd09b-803b-41d8-9fad-cdeb702dd43f"
/dev/sdk1: LABEL="swift-sdi1" UUID="145acd7d-d3a2-48de-8129-57440aa083a9" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdi1" PARTUUID="f552416a-8ccc-40cf-abd5-76182acbd73f"
/dev/sdd1: LABEL="swift-sdc1" UUID="37516634-0e19-47ed-8e10-1eac7145019f" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdc1" PARTUUID="c13f31af-6282-4929-9c6e-1d85369edbd2"
/dev/sdb1: UUID="90122c2e-ad6c-c49b-95c8-137cada4013f" UUID_SUB="abc61a09-4578-6a96-3312-a9944ad609de" LABEL="ms-be2056:0" TYPE="linux_raid_member" PARTUUID="0cdc45e1-01"
/dev/sdb2: UUID="50d4e233-8d1b-74b5-91a0-9bd491f987d2" UUID_SUB="b57661c5-c157-6958-7728-af9a4e69ae33" LABEL="ms-be2056:1" TYPE="linux_raid_member" PARTUUID="0cdc45e1-02"
/dev/sdb3: LABEL="swift-sdb3" UUID="f2bca9d7-f896-4dec-834b-fea69f0c4cf1" BLOCK_SIZE="4096" TYPE="xfs" PARTUUID="0cdc45e1-03"
/dev/sdb4: LABEL="swift-sdb4" UUID="7bc81d15-2135-4d09-a23d-f03f28c7f055" BLOCK_SIZE="4096" TYPE="xfs" PARTUUID="0cdc45e1-04"
/dev/sdc1: LABEL="swift-sdd1" UUID="a00571e1-46a6-435b-a943-476519c465f5" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="swift-sdd1" PARTUUID="6d9ef4c1-f5ba-4e37-9799-808fa0c79d6f"
/dev/md0: UUID="4267410e-7a5f-467e-b07d-b59c0349aefc" BLOCK_SIZE="4096" TYPE="ext4"
/dev/md1: UUID="84f6a334-7dec-4db6-bf86-0877e0b9ae1d" TYPE="swap"

...and now puppet runs OK

Related Objects

Mentioned In: T309027: Poweredge R730xd, R740xd, R740xd2 SSDs not visible to OS as SSDs
Mentioned Here: T309027: Poweredge R730xd, R740xd, R740xd2 SSDs not visible to OS as SSDs
T308677: unstable device mapping of SSDs causing installer problems - example reimage with destruction of swift filesystem

Event Timeline

MatthewVernon created this task.May 18 2022, 9:15 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 18 2022, 9:15 AM

LSobanski subscribed.May 18 2022, 9:24 AM

MatthewVernon mentioned this in T309027: Poweredge R730xd, R740xd, R740xd2 SSDs not visible to OS as SSDs.May 23 2022, 2:08 PM

T308677 shows an example where the installer destroys a filesystem.

Also related is that following T309027, all the SSDs on ms-* reliably appear as non-rotational, so could in theory be used to tell puppet (and, indeed, the installer) which drives are which.

@MatthewVernon is there still anything pending from I/F on this task or can be resolved in light of the follow up work done in related tasks and the cookbook?

I think we're at the point where we're going to move to the more reliable approach on a rolling basis as hardware gets replaced; so other than ironing out the remaining cookbook issues, probably not.

joanna_borun removed a project: Infrastructure-Foundations.Jan 29 2024, 4:18 PM

unstable device mapping of SSDs causing swift/puppet problems - example reimageOpen, Needs TriagePublicActions

Description

Related Objects

Event Timeline

unstable device mapping of SSDs causing swift/puppet problems - example reimage
Open, Needs TriagePublic
Actions