rack/setup/install new ms-be servers ms-be204[4-9] ,ms-be2050
Open, NormalPublic

Description

This task will track the racking, setup, and installation of 7 new session ms-be servers.

Racking Proposal:

ms-be2044 A2
ms-be2045 A7
ms-be2046 B2
ms-be2047 B7
ms-be2048 C2
ms-be2049 C7
ms-be2050 D7

ms-be2044

  • - receive in system on procurement task T201937
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan) asw-a2-codfw xe-2/0/4
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be2045

  • - receive in system on procurement task T201937
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan) asw-a7-codfw xe-7/0/0
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be2046

  • - receive in system on procurement task T201937
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan) asw-b2-codfw xe-2/0/4
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be2047

  • - receive in system on procurement task T201937
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan) asw-b7-codfw xe-7/0/0
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be2048

  • - receive in system on procurement task T201937
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan) asw-c2-codfw xe-2/0/2
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be2049

  • - receive in system on procurement task T201937
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan) asw-c7-codfw xe-7/0/0
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be2050

  • - receive in system on procurement task T201937
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan) asw-d7-codfw xe-7/0/8
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation
Papaul created this task.Tue, Nov 13, 5:10 PM
Restricted Application added a project: Operations. · View Herald TranscriptTue, Nov 13, 5:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Papaul triaged this task as Normal priority.Tue, Nov 13, 5:10 PM

Racking plan looks good to me!

Papaul updated the task description. (Show Details)Tue, Nov 13, 9:39 PM
Papaul updated the task description. (Show Details)Wed, Nov 14, 5:49 PM

Change 473646 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Add production and mgmt DNS entries for ms-be200[4-9] and ms-be2050

https://gerrit.wikimedia.org/r/473646

Papaul updated the task description. (Show Details)Thu, Nov 15, 12:29 AM
papaul@asw-a-codfw> show interfaces xe-2/0/4 descriptions 
Interface       Admin Link Description
xe-2/0/4        up    up   ms-be2044

papaul@asw-a-codfw# run show interfaces xe-7/0/0 descriptions        
Interface       Admin Link Description
xe-7/0/0        up    up   ms-be2045

papaul@asw-b-codfw# run show interfaces xe-2/0/4 descriptions 
Interface       Admin Link Description
xe-2/0/4        up    up   ms-be2046

papaul@asw-b-codfw# run show interfaces xe-7/0/0 descriptions        
Interface       Admin Link Description
xe-7/0/0        up    up   ms-be2047

papaul@asw-c-codfw# run show interfaces xe-2/0/2 descriptions 
Interface       Admin Link Description
xe-2/0/2        up    up   ms-be2048

papaul@asw-c-codfw# run show interfaces xe-7/0/0 descriptions 
Interface       Admin Link Description
xe-7/0/0        up    up   ms-be2049
Papaul updated the task description. (Show Details)Thu, Nov 15, 3:52 PM

Change 473810 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] Partman: Add new ms-be systems

https://gerrit.wikimedia.org/r/473810

Change 473817 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP: ADD MAC address entries for ms-be204[4-9] and ms-be2050

https://gerrit.wikimedia.org/r/473817

Change 473810 merged by Dzahn:
[operations/puppet@production] Partman: Add new ms-be systems

https://gerrit.wikimedia.org/r/473810

Change 473817 merged by Dzahn:
[operations/puppet@production] DHCP: ADD MAC address entries for ms-be204[4-9] and ms-be2050

https://gerrit.wikimedia.org/r/473817

Change 473646 merged by Dzahn:
[operations/dns@master] DNS: Add production and mgmt DNS entries for ms-be200[4-9] and ms-be2050

https://gerrit.wikimedia.org/r/473646

@fgiunchedi I did the install on the first system ms-be2044 please check the output below. If it looks good for you let me know so i can resume the install on the other systems. Thanks.

root@ms-be2044:~# fdisk -l
Disk /dev/sda: 446.6 GiB, 479559942144 bytes, 936640512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x3a489292

Device     Boot     Start       End   Sectors   Size Id Type
/dev/sda1            2048 117186559 117184512  55.9G fd Linux raid autodetect
/dev/sda2       117186560 119140351   1953792   954M fd Linux raid autodetect
/dev/sda3       119140352 314451967 195311616  93.1G 83 Linux
/dev/sda4       314451968 936638463 622186496 296.7G 83 Linux


Disk /dev/sdb: 446.6 GiB, 479559942144 bytes, 936640512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xb0a984f5

Device     Boot     Start       End   Sectors   Size Id Type
/dev/sdb1  *         2048 117186559 117184512  55.9G fd Linux raid autodetect
/dev/sdb2       117186560 119140351   1953792   954M fd Linux raid autodetect
/dev/sdb3       119140352 314451967 195311616  93.1G 83 Linux
/dev/sdb4       314451968 936638463 622186496 296.7G 83 Linux


Disk /dev/sdc: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdd: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
---
Papaul updated the task description. (Show Details)Fri, Nov 16, 7:05 PM
Papaul updated the task description. (Show Details)
Papaul updated the task description. (Show Details)Fri, Nov 16, 7:16 PM
Papaul updated the task description. (Show Details)Fri, Nov 16, 7:39 PM
Papaul updated the task description. (Show Details)
Papaul updated the task description. (Show Details)Fri, Nov 16, 9:10 PM

@fgiunchedi I did the install on the first system ms-be2044 please check the output below. If it looks good for you let me know so i can resume the install on the other systems. Thanks.

Thanks @Papaul ! Disk configuration looks good to me, I did a test reboot on ms-be2044 and came back as expected. You can resume with the other systems too with the same configuration. LMK once done and I'll start a quick burn-in/stress test.

Papaul updated the task description. (Show Details)Mon, Nov 19, 1:56 PM
Papaul updated the task description. (Show Details)Mon, Nov 19, 3:59 PM
Papaul reassigned this task from Papaul to fgiunchedi.

@fgiunchedi All yours

Change 474732 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] site: add ms-be205* hosts

https://gerrit.wikimedia.org/r/474732

Thanks @Papaul !

Writing down what I found and the fixes for reference

  • ms-be2046 doesn't show its spinning, only ssd. fixed with megacli -CfgEachDskRaid0 WB RA Direct CachedBadBBU -a0
  • ms-be2049 had sdn with four partitions not one, fixed with wipefs -a /dev/sdn1 + wipefs -a /dev/sdn
  • ms-be2050 didn't have the swift::backend role, fixed in puppet

Change 474732 merged by Filippo Giunchedi:
[operations/puppet@production] site: add ms-be205* hosts

https://gerrit.wikimedia.org/r/474732

Mentioned in SAL (#wikimedia-operations) [2018-11-20T09:23:05Z] <godog> stress-test new ms-be hardware - T209395

Change 475071 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: add new ms-be hosts

https://gerrit.wikimedia.org/r/475071

Change 475071 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: add new ms-be hosts

https://gerrit.wikimedia.org/r/475071

Mentioned in SAL (#wikimedia-operations) [2018-11-21T10:26:26Z] <godog> initial weight for new ms-be2* hosts (all but ms-be2047) - T209395

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.

Mentioned in SAL (#wikimedia-operations) [2018-11-27T08:15:41Z] <godog> more weight to new ms-be hosts in codfw - T209395

Mentioned in SAL (#wikimedia-operations) [2018-12-03T11:39:04Z] <godog> more weight to new ms-be codfw hosts - T209395

Mentioned in SAL (#wikimedia-operations) [2018-12-07T14:21:11Z] <godog> more weight to new ms-be codfw hosts - T209395

Mentioned in SAL (#wikimedia-operations) [2018-12-10T08:23:35Z] <godog> final round of weight addition to new ms-be codfw hosts - T209395