Page MenuHomePhabricator
Create Task
Maniphest T209395

rack/setup/install new ms-be servers ms-be204[4-9] ,ms-be2050
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and installation of 7 new session ms-be servers.

Racking Proposal:

ms-be2044 A2
ms-be2045 A7
ms-be2046 B2
ms-be2047 B7
ms-be2048 C2
ms-be2049 C7
ms-be2050 D7

ms-be2044

  • - receive in system on procurement task T201937
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan) asw-a2-codfw xe-2/0/4
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be2045

  • - receive in system on procurement task T201937
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan) asw-a7-codfw xe-7/0/0
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be2046

  • - receive in system on procurement task T201937
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan) asw-b2-codfw xe-2/0/4
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be2047

  • - receive in system on procurement task T201937
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan) asw-b7-codfw xe-7/0/0
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be2048

  • - receive in system on procurement task T201937
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan) asw-c2-codfw xe-2/0/2
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be2049

  • - receive in system on procurement task T201937
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan) asw-c7-codfw xe-7/0/0
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be2050

  • - receive in system on procurement task T201937
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, internal vlan) asw-d7-codfw xe-7/0/8
    • end on-site specific steps
  • - production dns entries added (internal subnets for each row)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

Details

SubjectRepoBranchLines +/-
hieradata: add new ms-be hostsoperations/puppetproduction+7 -0
site: add ms-be205* hostsoperations/puppetproduction+1 -1
DNS: Add production and mgmt DNS entries for ms-be200[4-9] and ms-be2050operations/dnsmaster+42 -2
DHCP: ADD MAC address entries for ms-be204[4-9] and ms-be2050operations/puppetproduction+35 -0
Partman: Add new ms-be systemsoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Papaul created this task.Nov 13 2018, 5:10 PM
Restricted Application added a project: SRE. · View Herald TranscriptNov 13 2018, 5:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Papaul triaged this task as Medium priority.Nov 13 2018, 5:10 PM
fgiunchedi added a comment.Nov 13 2018, 5:17 PM

Racking plan looks good to me!

Papaul updated the task description. (Show Details)Nov 13 2018, 9:39 PM
Papaul updated the task description. (Show Details)Nov 14 2018, 5:49 PM
gerritbot subscribed.Nov 15 2018, 12:20 AM

Change 473646 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Add production and mgmt DNS entries for ms-be200[4-9] and ms-be2050

https://gerrit.wikimedia.org/r/473646

gerritbot added a project: Patch-For-Review.Nov 15 2018, 12:20 AM
Papaul updated the task description. (Show Details)Nov 15 2018, 12:29 AM
Papaul added a comment.Nov 15 2018, 3:51 PM
papaul@asw-a-codfw> show interfaces xe-2/0/4 descriptions 
Interface       Admin Link Description
xe-2/0/4        up    up   ms-be2044

papaul@asw-a-codfw# run show interfaces xe-7/0/0 descriptions        
Interface       Admin Link Description
xe-7/0/0        up    up   ms-be2045

papaul@asw-b-codfw# run show interfaces xe-2/0/4 descriptions 
Interface       Admin Link Description
xe-2/0/4        up    up   ms-be2046

papaul@asw-b-codfw# run show interfaces xe-7/0/0 descriptions        
Interface       Admin Link Description
xe-7/0/0        up    up   ms-be2047

papaul@asw-c-codfw# run show interfaces xe-2/0/2 descriptions 
Interface       Admin Link Description
xe-2/0/2        up    up   ms-be2048

papaul@asw-c-codfw# run show interfaces xe-7/0/0 descriptions 
Interface       Admin Link Description
xe-7/0/0        up    up   ms-be2049
Papaul updated the task description. (Show Details)Nov 15 2018, 3:52 PM
gerritbot added a comment.Nov 15 2018, 6:28 PM

Change 473810 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] Partman: Add new ms-be systems

https://gerrit.wikimedia.org/r/473810

gerritbot added a comment.Nov 15 2018, 6:52 PM

Change 473817 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP: ADD MAC address entries for ms-be204[4-9] and ms-be2050

https://gerrit.wikimedia.org/r/473817

gerritbot added a comment.Nov 16 2018, 4:00 PM

Change 473810 merged by Dzahn:
[operations/puppet@production] Partman: Add new ms-be systems

https://gerrit.wikimedia.org/r/473810

gerritbot added a comment.Nov 16 2018, 4:10 PM

Change 473817 merged by Dzahn:
[operations/puppet@production] DHCP: ADD MAC address entries for ms-be204[4-9] and ms-be2050

https://gerrit.wikimedia.org/r/473817

gerritbot added a comment.Nov 16 2018, 4:21 PM

Change 473646 merged by Dzahn:
[operations/dns@master] DNS: Add production and mgmt DNS entries for ms-be200[4-9] and ms-be2050

https://gerrit.wikimedia.org/r/473646

Papaul added a comment.Nov 16 2018, 5:00 PM

@fgiunchedi I did the install on the first system ms-be2044 please check the output below. If it looks good for you let me know so i can resume the install on the other systems. Thanks.

root@ms-be2044:~# fdisk -l
Disk /dev/sda: 446.6 GiB, 479559942144 bytes, 936640512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x3a489292

Device     Boot     Start       End   Sectors   Size Id Type
/dev/sda1            2048 117186559 117184512  55.9G fd Linux raid autodetect
/dev/sda2       117186560 119140351   1953792   954M fd Linux raid autodetect
/dev/sda3       119140352 314451967 195311616  93.1G 83 Linux
/dev/sda4       314451968 936638463 622186496 296.7G 83 Linux


Disk /dev/sdb: 446.6 GiB, 479559942144 bytes, 936640512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xb0a984f5

Device     Boot     Start       End   Sectors   Size Id Type
/dev/sdb1  *         2048 117186559 117184512  55.9G fd Linux raid autodetect
/dev/sdb2       117186560 119140351   1953792   954M fd Linux raid autodetect
/dev/sdb3       119140352 314451967 195311616  93.1G 83 Linux
/dev/sdb4       314451968 936638463 622186496 296.7G 83 Linux


Disk /dev/sdc: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdd: 3.7 TiB, 4000225165312 bytes, 7812939776 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
---
Papaul updated the task description. (Show Details)Nov 16 2018, 7:05 PM
Papaul updated the task description. (Show Details)
Papaul updated the task description. (Show Details)Nov 16 2018, 7:16 PM
Papaul updated the task description. (Show Details)Nov 16 2018, 7:39 PM
Papaul updated the task description. (Show Details)
Papaul updated the task description. (Show Details)Nov 16 2018, 9:10 PM
fgiunchedi added a comment.Nov 19 2018, 8:41 AM
In T209395#4754151, @Papaul wrote:

@fgiunchedi I did the install on the first system ms-be2044 please check the output below. If it looks good for you let me know so i can resume the install on the other systems. Thanks.

Thanks @Papaul ! Disk configuration looks good to me, I did a test reboot on ms-be2044 and came back as expected. You can resume with the other systems too with the same configuration. LMK once done and I'll start a quick burn-in/stress test.

fgiunchedi mentioned this in T209727: Degraded RAID on ms-be2046.Nov 19 2018, 8:41 AM
Papaul updated the task description. (Show Details)Nov 19 2018, 1:56 PM
Papaul reassigned this task from Papaul to fgiunchedi.Nov 19 2018, 3:59 PM
Papaul updated the task description. (Show Details)

@fgiunchedi All yours

gerritbot added a comment.Nov 19 2018, 4:22 PM

Change 474732 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] site: add ms-be205* hosts

https://gerrit.wikimedia.org/r/474732

fgiunchedi added a comment.Nov 19 2018, 4:25 PM

Thanks @Papaul !

Writing down what I found and the fixes for reference

  • ms-be2046 doesn't show its spinning, only ssd. fixed with megacli -CfgEachDskRaid0 WB RA Direct CachedBadBBU -a0
  • ms-be2049 had sdn with four partitions not one, fixed with wipefs -a /dev/sdn1 + wipefs -a /dev/sdn
  • ms-be2050 didn't have the swift::backend role, fixed in puppet
gerritbot added a comment.Nov 19 2018, 4:26 PM

Change 474732 merged by Filippo Giunchedi:
[operations/puppet@production] site: add ms-be205* hosts

https://gerrit.wikimedia.org/r/474732

Stashbot subscribed.Nov 20 2018, 9:23 AM

Mentioned in SAL (#wikimedia-operations) [2018-11-20T09:23:05Z] <godog> stress-test new ms-be hardware - T209395

gerritbot added a comment.Nov 21 2018, 9:42 AM

Change 475071 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: add new ms-be hosts

https://gerrit.wikimedia.org/r/475071

gerritbot added a comment.Nov 21 2018, 9:44 AM

Change 475071 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: add new ms-be hosts

https://gerrit.wikimedia.org/r/475071

Stashbot added a comment.Nov 21 2018, 10:26 AM

Mentioned in SAL (#wikimedia-operations) [2018-11-21T10:26:26Z] <godog> initial weight for new ms-be2* hosts (all but ms-be2047) - T209395

fgiunchedi added a project: User-fgiunchedi.Nov 21 2018, 1:18 PM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.
Stashbot added a comment.Nov 27 2018, 8:15 AM

Mentioned in SAL (#wikimedia-operations) [2018-11-27T08:15:41Z] <godog> more weight to new ms-be hosts in codfw - T209395

Marostegui mentioned this in T210697: ms-be2047 rebooting itself.Nov 29 2018, 7:11 AM
Stashbot added a comment.Dec 3 2018, 11:39 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-03T11:39:04Z] <godog> more weight to new ms-be codfw hosts - T209395

Stashbot added a comment.Dec 7 2018, 2:21 PM

Mentioned in SAL (#wikimedia-operations) [2018-12-07T14:21:11Z] <godog> more weight to new ms-be codfw hosts - T209395

Stashbot added a comment.Dec 10 2018, 8:23 AM

Mentioned in SAL (#wikimedia-operations) [2018-12-10T08:23:35Z] <godog> final round of weight addition to new ms-be codfw hosts - T209395

fgiunchedi closed this task as Resolved.Dec 13 2018, 9:02 AM

This is completed, modulo ms-be2047 being diagnosed in T209921

Stashbot added a comment.Feb 5 2019, 1:55 PM

Mentioned in SAL (#wikimedia-operations) [2019-02-05T13:55:15Z] <godog> swift codfw-prod: add ms-be2047 - T209395 T209921

Stashbot mentioned this in T209921: ms-be2047 spontaneous reboots.Feb 5 2019, 1:55 PM
Stashbot added a comment.Feb 6 2019, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-06T09:15:42Z] <godog> swift codfw-prod: more weight for ms-be2047 - T209395 T209921

Stashbot added a comment.Feb 7 2019, 8:34 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-07T08:34:24Z] <godog> swift codfw-prod: more weight to ms-be2047 - T209395 T209921

Stashbot added a comment.Feb 8 2019, 10:23 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-08T10:23:38Z] <godog> swift codfw-prod: more weight to ms-be2047 - T209395 T209921

Papaul closed subtask T209921: ms-be2047 spontaneous reboots as Resolved.Feb 11 2019, 3:49 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits