Page MenuHomePhabricator

Rack and setup ms-be1040-1043
Closed, DuplicatePublic

Description

@fgiunchedi These servers are racked in 10G racks a7,b7,c7 and d2 (d7 has several ms-be servers already)

ms-be1040:

  • - receive in system on procurement task T187383
  • - rack system & update racktables
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private vlan)
    • end on-site specific steps
  • - production dns entries added (private subnet)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be1041:

  • - receive in system on procurement task T187383
  • - rack system & update racktables
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private vlan)
    • end on-site specific steps
  • - production dns entries added (private subnet)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be1042:

  • - receive in system on procurement task T187383
  • - rack system & update racktables
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private vlan)
    • end on-site specific steps
  • - production dns entries added (private subnet)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be1043:

  • - receive in system on procurement task T187383
  • - rack system & update racktables
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private vlan)
    • end on-site specific steps
  • - production dns entries added (private subnet)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 10 2018, 2:10 PM

Change 425274 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding dns entries for ms-be1040-43

https://gerrit.wikimedia.org/r/425274

Change 425274 merged by Cmjohnson:
[operations/dns@master] Adding dns entries for ms-be1040-43

https://gerrit.wikimedia.org/r/425274

Cmjohnson updated the task description. (Show Details)Apr 10 2018, 5:11 PM

@ayounsi These are racked in 10G racks and I would like to utilize the new switches....Can you assign ports please

1040 A7 u29/30 ( probably something in the xe-7/0/30's)
1041 B7 u18/19 probably something in the xe-7/0/15-22 range)
1042 C7 u30/31 ( probably something in the xe-7/0/30's)
1043 D2 ( I will take care of this)

Thanks!

@ayounsi I cabled the new ms-be servers to the following. Please let me know if you want that changed.

1040 a7 xe-7/0/28
1041 b7 xe-7/0/18
1042 c7 xe-7/0/28
1043 d2 xe-2/0/3

Cmjohnson updated the task description. (Show Details)Apr 17 2018, 2:56 PM

@ayounsi I cabled the new ms-be servers to the following. Please let me know if you want that changed.
1040 a7 xe-7/0/28
1041 b7 xe-7/0/18
1042 c7 xe-7/0/28
1043 d2 xe-2/0/3

asw-a-eqiad:ge-7/0/28 already have analytics1071. Please use 7/0/29. Other than that it's fine.
To make sure we don't double book ports, please update the description of the matching port on asw to reflect what's on asw2 (eg. ms-be1040 on asw2)

Configured the raid, set to all 14 disks are raid 0 with the ssd in slot 12 first and the ssd in slot 13 second, the other 12 disks are set to raid 0 in order.

Updated the switch ports for both asw-x-eqiad and asw2-x-eqiad for the new ms-be servers and moved the ms-be1042 to xe-7/0/29 as requested. I was not able to set the vlans for for ms-be1040 and ms-be1042.

@ayounsi Can you import the vlans to asw-a & c-eqiad?

Private interfaces ranges have been created on asw2-a/c-eqiad and interfaces added (they can't be created without interfaces).

ayounsi updated the task description. (Show Details)Apr 18 2018, 5:41 PM
fgiunchedi added a parent task: Unknown Object (Task).Apr 23 2018, 12:53 PM

Change 428339 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding dhcpd file for ms-be104-43

https://gerrit.wikimedia.org/r/428339

Change 428339 merged by Cmjohnson:
[operations/puppet@production] Adding dhcpd file for ms-be104-43

https://gerrit.wikimedia.org/r/428339

Cmjohnson updated the task description. (Show Details)Apr 23 2018, 2:08 PM

@Marostegui yes raid is setup to raid 10 256K stripe

@Marostegui yes raid is setup to raid 10 256K stripe

I guess this is for: T191792 :)

Cmjohnson reassigned this task from Cmjohnson to fgiunchedi.Apr 23 2018, 8:36 PM
Cmjohnson updated the task description. (Show Details)
Cmjohnson removed a project: ops-eqiad.

@fgiunchedi These are all yours for implementation.

Change 428572 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] netops: add asw2-a-eqiad and asw2-c-eqiad

https://gerrit.wikimedia.org/r/428572

Change 428572 merged by Filippo Giunchedi:
[operations/puppet@production] netops: add asw2-a-eqiad and asw2-c-eqiad

https://gerrit.wikimedia.org/r/428572

Looks like 3 out of 4 hosts have sda or sdb as one of the HDDs, not SSDs. The remaining host has sda/sdb as SSDs and two additional mdadm raid arrays.

@Cmjohnson anything in the setup that was done differently on ms-be1043 (with the right config, sda+sdb as ssd) compared to the rest?

root@neodymium:~# cumin 'ms-be104*' 'grep sd[ab]$ /proc/partitions | sort'
4 hosts will be targeted:
ms-be[1040-1043].eqiad.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====                                                                                                             
(2) ms-be[1040-1041].eqiad.wmnet                                                                                                   
----- OUTPUT of 'grep sd[ab]$ /pr...artitions | sort' -----                                                                        
   8        0 3906469888 sda                                                                                                       
   8       16  468320256 sdb                                                                                                       
===== NODE GROUP =====                                                                                                             
(1) ms-be1042.eqiad.wmnet                                                                                                          
----- OUTPUT of 'grep sd[ab]$ /pr...artitions | sort' -----                                                                        
   8        0  468320256 sda                                                                                                       
   8       16 3906469888 sdb                                                                                                       
===== NODE GROUP =====                                                                                                             
(1) ms-be1043.eqiad.wmnet                                                                                                          
----- OUTPUT of 'grep sd[ab]$ /pr...artitions | sort' -----                                                                        
   8        0  468320256 sda                                                                                                       
   8       16  468320256 sdb

@Cmjohnson confirmed raid config is the same on all of those, I rebooted the hosts showing the incorrect order and indeed upon reboot the order is as expected:

===== NODE GROUP =====                                                                                                             
(4) ms-be[1040-1043].eqiad.wmnet                                                                                                   
----- OUTPUT of 'grep sd[ab]$ /pr...artitions | sort' -----                                                                        
   8        0  468320256 sda                                                                                                       
   8       16  468320256 sdb

Change 428661 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] Add ms-be104[0123] to swift::storage

https://gerrit.wikimedia.org/r/428661

Change 428661 merged by Filippo Giunchedi:
[operations/puppet@production] Add ms-be104[0123] to swift::storage

https://gerrit.wikimedia.org/r/428661

Mentioned in SAL (#wikimedia-operations) [2018-04-30T08:23:21Z] <godog> swift eqiad-prod more weight to ms-be104[0-3] - T191896