Page MenuHomePhabricator

lvs2009/lvs2010 with no RAID configured
Closed, ResolvedPublic

Description

As reported by @faidon in T136562, lvs2009 and lvs2010 lack proper RAID configuration:

lvs2009
vgutierrez@lvs2009:~$ sudo fdisk -l
Disk /dev/sda: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xf47ed367

Device     Boot     Start       End   Sectors   Size Id Type
/dev/sda1  *         2048 466862079 466860032 222.6G 83 Linux
/dev/sda2       466864126 468860927   1996802   975M  5 Extended
/dev/sda5       466864128 468860927   1996800   975M 82 Linux swap / Solaris

Partition 2 does not start on physical sector boundary.


Disk /dev/sdb: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 847DAEB4-B413-48A4-8059-EFEA1A07EC78

Device     Start       End   Sectors   Size Type
/dev/sdb1   2048 468860927 468858880 223.6G Microsoft basic data
lvs2010
vgutierrez@lvs2010:~$ sudo fdisk -l
Disk /dev/sda: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xefe402d4

Device     Boot     Start       End   Sectors   Size Id Type
/dev/sda1  *         2048 466862079 466860032 222.6G 83 Linux
/dev/sda2       466864126 468860927   1996802   975M  5 Extended
/dev/sda5       466864128 468860927   1996800   975M 82 Linux swap / Solaris

Partition 2 does not start on physical sector boundary.


Disk /dev/sdb: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 1DDD8559-9054-491B-A640-9FFEDA68E0B5

Device     Start       End   Sectors   Size Type
/dev/sdb1   2048 468860927 468858880 223.6G Microsoft basic data

Event Timeline

Vgutierrez triaged this task as High priority.Oct 2 2018, 12:32 PM
Vgutierrez created this task.
Vgutierrez moved this task from Triage to Hardware on the Traffic board.

Looks like the culprit is https://github.com/wikimedia/puppet/blob/production/modules/install_server/files/autoinstall/netboot.cfg#L126-L127:

lvs100[7-9]|lvs101[012]|lvs2*) echo partman/flat.cfg ;; \
lvs100[1-6]|lvs101[3-6]|lvs[345]*|lvs2009|lvs2010) echo partman/raid1-lvm.cfg ;; \

i would say this is part of T196560 which is still open. i left a comment there.

Papaul added a subscriber: Papaul.Oct 2 2018, 5:13 PM

@Vgutierrez does this mean reinstalling both LVS servers (lvs2009 and lvs2010) if yes please elaborate how you want to approach this .

Thanks.

@Papaul that's right, reinstalling both servers would be the fastest/safest approach :)

Papaul added a comment.Oct 3 2018, 2:35 PM

@Vgutierrez let me know when I have green light to start working on this.

Change 464156 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] install_server: limit flat.cfg to lvs200[1-6] letting lvs20[09-10] out

https://gerrit.wikimedia.org/r/464156

Change 464156 merged by Dzahn:
[operations/puppet@production] install_server: limit flat.cfg to lvs200[1-6] letting lvs20[09-10] out

https://gerrit.wikimedia.org/r/464156

Papaul added a comment.Oct 3 2018, 5:10 PM

puppet certs for lvs2009 and lvs2010 delete from master for OS reinstall

Papaul added a comment.EditedOct 3 2018, 6:18 PM

The reason the OS reinstall is taking long is that when all 4 NIC's are plugged in, the server can not auto configure the first NIC so I have to go in the BIOS and disable the other 3 NIC's. The OS is installing now.

Papaul added a comment.Oct 3 2018, 6:30 PM

LVS 2009

root@lvs2009:~# fdisk -l
Disk /dev/sda: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x2c601a25

Device     Boot    Start       End   Sectors  Size Id Type
/dev/sda1  *        2048  97656831  97654784 46.6G fd Linux raid autodetect
/dev/sda2       97656832 468860927 371204096  177G fd Linux raid autodetect


Disk /dev/sdb: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x46a3abf2

Device     Boot    Start       End   Sectors  Size Id Type
/dev/sdb1           2048  97656831  97654784 46.6G fd Linux raid autodetect
/dev/sdb2       97656832 468860927 371204096  177G fd Linux raid autodetect


Disk /dev/md1: 176.9 GiB, 189922279424 bytes, 370941952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/md0: 46.5 GiB, 49965694976 bytes, 97589248 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/lvs2009--vg-swap: 952 MiB, 998244352 bytes, 1949696 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/lvs2009--vg-_placeholder: 176 GiB, 188919840768 bytes, 368984064 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Papaul added a comment.Oct 3 2018, 6:35 PM
  • Re-enable the other 3 NIC's
  • First puppet run complete

LVS2009 is ready

Papaul added a comment.Oct 3 2018, 7:05 PM

lvs2010

root@lvs2010:~# fdisk -l
Disk /dev/sda: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x8a6958cc

Device     Boot    Start       End   Sectors  Size Id Type
/dev/sda1  *        2048  97656831  97654784 46.6G fd Linux raid autodetect
/dev/sda2       97656832 468860927 371204096  177G fd Linux raid autodetect


Disk /dev/sdb: 223.6 GiB, 240057409536 bytes, 468862128 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xdd5dea9a

Device     Boot    Start       End   Sectors  Size Id Type
/dev/sdb1           2048  97656831  97654784 46.6G fd Linux raid autodetect
/dev/sdb2       97656832 468860927 371204096  177G fd Linux raid autodetect


Disk /dev/md0: 46.5 GiB, 49965694976 bytes, 97589248 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/md1: 176.9 GiB, 189922279424 bytes, 370941952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/lvs2010--vg-swap: 952 MiB, 998244352 bytes, 1949696 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mapper/lvs2010--vg-_placeholder: 176 GiB, 188919840768 bytes, 368984064 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Lvs2010 is ready.

Vgutierrez closed this task as Resolved.Oct 3 2018, 7:08 PM
Vgutierrez assigned this task to Papaul.
Vgutierrez removed a project: Patch-For-Review.