Page MenuHomePhabricator
Create Task
Maniphest T191896

Rack and setup ms-be1040-1043
Closed, DuplicatePublic
Actions

Description

@fgiunchedi These servers are racked in 10G racks a7,b7,c7 and d2 (d7 has several ms-be servers already)

ms-be1040:

  • - receive in system on procurement task T187383
  • - rack system & update racktables
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private vlan)
    • end on-site specific steps
  • - production dns entries added (private subnet)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be1041:

  • - receive in system on procurement task T187383
  • - rack system & update racktables
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private vlan)
    • end on-site specific steps
  • - production dns entries added (private subnet)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be1042:

  • - receive in system on procurement task T187383
  • - rack system & update racktables
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private vlan)
    • end on-site specific steps
  • - production dns entries added (private subnet)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

ms-be1043:

  • - receive in system on procurement task T187383
  • - rack system & update racktables
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private vlan)
    • end on-site specific steps
  • - production dns entries added (private subnet)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch)
  • - puppet accept/initial run
  • - handoff for service implementation

Details

SubjectRepoBranchLines +/-
Add ms-be104[0123] to swift::storageoperations/puppetproduction+8 -4
netops: add asw2-a-eqiad and asw2-c-eqiadoperations/puppetproduction+2 -0
Adding dhcpd file for ms-be104-43operations/puppetproduction+20 -0
Adding dns entries for ms-be1040-43operations/dnsmaster+25 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 DuplicatefgiunchediT191896 Rack and setup ms-be1040-1043

Event Timeline

• Cmjohnson created this task.Apr 10 2018, 2:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 10 2018, 2:10 PM
gerritbot subscribed.Apr 10 2018, 2:25 PM

Change 425274 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding dns entries for ms-be1040-43

https://gerrit.wikimedia.org/r/425274

gerritbot added a project: Patch-For-Review.Apr 10 2018, 2:25 PM
• Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.Apr 10 2018, 2:28 PM
gerritbot added a comment.Apr 10 2018, 4:34 PM

Change 425274 merged by Cmjohnson:
[operations/dns@master] Adding dns entries for ms-be1040-43

https://gerrit.wikimedia.org/r/425274

• Cmjohnson updated the task description. (Show Details)Apr 10 2018, 5:11 PM
• Cmjohnson added a subscriber: ayounsi.Apr 10 2018, 5:14 PM

@ayounsi These are racked in 10G racks and I would like to utilize the new switches....Can you assign ports please

1040 A7 u29/30 ( probably something in the xe-7/0/30's)
1041 B7 u18/19 probably something in the xe-7/0/15-22 range)
1042 C7 u30/31 ( probably something in the xe-7/0/30's)
1043 D2 ( I will take care of this)

Thanks!

• Cmjohnson added a comment.Apr 17 2018, 2:56 PM

@ayounsi I cabled the new ms-be servers to the following. Please let me know if you want that changed.

1040 a7 xe-7/0/28
1041 b7 xe-7/0/18
1042 c7 xe-7/0/28
1043 d2 xe-2/0/3

• Cmjohnson updated the task description. (Show Details)Apr 17 2018, 2:56 PM
ayounsi added a comment.Apr 17 2018, 6:32 PM
In T191896#4136243, @Cmjohnson wrote:

@ayounsi I cabled the new ms-be servers to the following. Please let me know if you want that changed.

1040 a7 xe-7/0/28
1041 b7 xe-7/0/18
1042 c7 xe-7/0/28
1043 d2 xe-2/0/3

asw-a-eqiad:ge-7/0/28 already have analytics1071. Please use 7/0/29. Other than that it's fine.
To make sure we don't double book ports, please update the description of the matching port on asw to reflect what's on asw2 (eg. ms-be1040 on asw2)

• Cmjohnson added a comment.Apr 18 2018, 3:20 PM

Configured the raid, set to all 14 disks are raid 0 with the ssd in slot 12 first and the ssd in slot 13 second, the other 12 disks are set to raid 0 in order.

Updated the switch ports for both asw-x-eqiad and asw2-x-eqiad for the new ms-be servers and moved the ms-be1042 to xe-7/0/29 as requested. I was not able to set the vlans for for ms-be1040 and ms-be1042.

@ayounsi Can you import the vlans to asw-a & c-eqiad?

ayounsi added a comment.Apr 18 2018, 5:40 PM

Private interfaces ranges have been created on asw2-a/c-eqiad and interfaces added (they can't be created without interfaces).

ayounsi updated the task description. (Show Details)Apr 18 2018, 5:41 PM
fgiunchedi added a parent task: Unknown Object (Task).Apr 23 2018, 12:53 PM
gerritbot added a comment.Apr 23 2018, 2:04 PM

Change 428339 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Adding dhcpd file for ms-be104-43

https://gerrit.wikimedia.org/r/428339

gerritbot added a comment.Apr 23 2018, 2:06 PM

Change 428339 merged by Cmjohnson:
[operations/puppet@production] Adding dhcpd file for ms-be104-43

https://gerrit.wikimedia.org/r/428339

• Cmjohnson updated the task description. (Show Details)Apr 23 2018, 2:08 PM
• Cmjohnson added a subscriber: • Marostegui.Apr 23 2018, 3:55 PM

@Marostegui yes raid is setup to raid 10 256K stripe

• Marostegui added a comment.Apr 23 2018, 3:56 PM
In T191896#4150295, @Cmjohnson wrote:

@Marostegui yes raid is setup to raid 10 256K stripe

I guess this is for: T191792 :)

• Cmjohnson reassigned this task from • Cmjohnson to fgiunchedi.Apr 23 2018, 8:36 PM
• Cmjohnson updated the task description. (Show Details)
• Cmjohnson removed a project: ops-eqiad.

@fgiunchedi These are all yours for implementation.

gerritbot added a comment.Apr 24 2018, 8:22 AM

Change 428572 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] netops: add asw2-a-eqiad and asw2-c-eqiad

https://gerrit.wikimedia.org/r/428572

gerritbot added a comment.Apr 24 2018, 8:52 AM

Change 428572 merged by Filippo Giunchedi:
[operations/puppet@production] netops: add asw2-a-eqiad and asw2-c-eqiad

https://gerrit.wikimedia.org/r/428572

fgiunchedi added a comment.Apr 24 2018, 1:26 PM

Looks like 3 out of 4 hosts have sda or sdb as one of the HDDs, not SSDs. The remaining host has sda/sdb as SSDs and two additional mdadm raid arrays.

@Cmjohnson anything in the setup that was done differently on ms-be1043 (with the right config, sda+sdb as ssd) compared to the rest?

root@neodymium:~# cumin 'ms-be104*' 'grep sd[ab]$ /proc/partitions | sort'
4 hosts will be targeted:
ms-be[1040-1043].eqiad.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====                                                                                                             
(2) ms-be[1040-1041].eqiad.wmnet                                                                                                   
----- OUTPUT of 'grep sd[ab]$ /pr...artitions | sort' -----                                                                        
   8        0 3906469888 sda                                                                                                       
   8       16  468320256 sdb                                                                                                       
===== NODE GROUP =====                                                                                                             
(1) ms-be1042.eqiad.wmnet                                                                                                          
----- OUTPUT of 'grep sd[ab]$ /pr...artitions | sort' -----                                                                        
   8        0  468320256 sda                                                                                                       
   8       16 3906469888 sdb                                                                                                       
===== NODE GROUP =====                                                                                                             
(1) ms-be1043.eqiad.wmnet                                                                                                          
----- OUTPUT of 'grep sd[ab]$ /pr...artitions | sort' -----                                                                        
   8        0  468320256 sda                                                                                                       
   8       16  468320256 sdb
fgiunchedi mentioned this in T192874: Degraded RAID on ms-be1043.Apr 24 2018, 1:42 PM
fgiunchedi added a comment.Apr 24 2018, 3:17 PM

@Cmjohnson confirmed raid config is the same on all of those, I rebooted the hosts showing the incorrect order and indeed upon reboot the order is as expected:

===== NODE GROUP =====                                                                                                             
(4) ms-be[1040-1043].eqiad.wmnet                                                                                                   
----- OUTPUT of 'grep sd[ab]$ /pr...artitions | sort' -----                                                                        
   8        0  468320256 sda                                                                                                       
   8       16  468320256 sdb
gerritbot added a comment.Apr 24 2018, 3:22 PM

Change 428661 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] Add ms-be104[0123] to swift::storage

https://gerrit.wikimedia.org/r/428661

gerritbot added a comment.Apr 24 2018, 3:25 PM

Change 428661 merged by Filippo Giunchedi:
[operations/puppet@production] Add ms-be104[0123] to swift::storage

https://gerrit.wikimedia.org/r/428661

fgiunchedi closed this task as a duplicate of T190081: rack/setup/install ms-be104[0-3].eqiad.wmnet.Apr 25 2018, 7:59 AM
Stashbot subscribed.Apr 30 2018, 8:23 AM

Mentioned in SAL (#wikimedia-operations) [2018-04-30T08:23:21Z] <godog> swift eqiad-prod more weight to ms-be104[0-3] - T191896

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL