Page MenuHomePhabricator

rack/setup/install ms-be205[1-6].codfw.wmnet
Closed, ResolvedPublic

Description

This task will track the racking and setup of 6 new ms-be systems in codfw.

This is expanding the backend service, not replacing any existing servers.

Hostnames: ms-be205[1-6].codfw.wmnet
Racking Proposal: 10G racks, 2x hosts in rows A+C. 1x host in rows B+D. Rack diversity with existing ms-be hw is preferred but not required.
Networking/Subnet/VLAN/IP: Internal subnet per row
Partitioning/Raid: use existing ms-be setup

ms-be2051:

  • - receive in system on procurement task T228460
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be2052:

  • - receive in system on procurement task T228460
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be2053:

  • - receive in system on procurement task T228460
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be2054:

  • - receive in system on procurement task T228460
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be2055:

  • - receive in system on procurement task T228460
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be2056:

  • - receive in system on procurement task T228460
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

Details

Related Gerrit Patches:
operations/software/swift-ring : mastercodfw-prod: add ms-be2051, minimal weight and servers_per_port
operations/puppet : productionhieradata: add ms-be205[1-6]
operations/dns : masterMove ms-be2055 to row C
operations/puppet : productionDHCP: Add MAC address entries for ms-be205[1-6]
operations/dns : masterDNS: Add mgmt and production DNS for ms-be205[1-6]

Event Timeline

RobH triaged this task as Medium priority.Sep 23 2019, 4:25 PM
RobH created this task.
Restricted Application added a project: Operations. · View Herald TranscriptSep 23 2019, 4:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
RobH added a parent task: Unknown Object (Task).Sep 23 2019, 4:25 PM
Papaul updated the task description. (Show Details)Sep 24 2019, 4:03 PM
Papaul updated the task description. (Show Details)Sep 24 2019, 9:31 PM
Papaul updated the task description. (Show Details)Sep 25 2019, 2:01 PM
Papaul updated the task description. (Show Details)Sep 25 2019, 2:25 PM
Papaul moved this task from Backlog to Racking Tasks on the ops-codfw board.
Papaul updated the task description. (Show Details)Sep 26 2019, 3:50 PM

Change 539412 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Add mgmt and production DNS for ms-be205[1-6]

https://gerrit.wikimedia.org/r/539412

Change 539412 merged by Papaul:
[operations/dns@master] DNS: Add mgmt and production DNS for ms-be205[1-6]

https://gerrit.wikimedia.org/r/539412

Papaul updated the task description. (Show Details)Sep 26 2019, 8:40 PM
Papaul updated the task description. (Show Details)Sep 26 2019, 9:00 PM

Change 539432 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP: Add MAC address entries for ms-be205[1-6]

https://gerrit.wikimedia.org/r/539432

Change 539432 merged by Papaul:
[operations/puppet@production] DHCP: Add MAC address entries for ms-be205[1-6]

https://gerrit.wikimedia.org/r/539432

Papaul updated the task description. (Show Details)Sep 27 2019, 2:08 PM
Papaul updated the task description. (Show Details)Sep 27 2019, 3:06 PM
Papaul updated the task description. (Show Details)Sep 27 2019, 4:06 PM
Papaul updated the task description. (Show Details)Sep 27 2019, 5:06 PM
Papaul updated the task description. (Show Details)Sep 27 2019, 5:46 PM
RobH removed a subscriber: RobH.Sep 27 2019, 6:00 PM
Papaul updated the task description. (Show Details)Sep 27 2019, 6:04 PM
Papaul reassigned this task from Papaul to fgiunchedi.Sep 27 2019, 10:32 PM
Papaul updated the task description. (Show Details)
Papaul added a subscriber: Papaul.

@fgiunchedi all yours

fgiunchedi reassigned this task from fgiunchedi to Papaul.Sep 30 2019, 1:28 PM

@fgiunchedi all yours

Thanks @Papaul !

I see two hosts in row A+D per netbox, and one in B+C, although it should be two hosts in A+C and one in B+D. In other words there's a misplace host in row D that should be in row C instead.

@sorry I fogot to mentioned that on the task, It was the faster and easier way to rack those servers. We can still move 1 host from D to C but it will take a while for me to get in into C2 if you find with that.

If you want to move 1 host from D to C please change the IP address of ms-be2055 from 10.192.48.X/22 to 10 192.32.X/22 and power it down

Thanks.

Change 539982 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] Move ms-be2055 to row C

https://gerrit.wikimedia.org/r/539982

Change 539982 merged by Filippo Giunchedi:
[operations/dns@master] Move ms-be2055 to row C

https://gerrit.wikimedia.org/r/539982

@sorry I fogot to mentioned that on the task, It was the faster and easier way to rack those servers. We can still move 1 host from D to C but it will take a while for me to get in into C2 if you find with that.
If you want to move 1 host from D to C please change the IP address of ms-be2055 from 10.192.48.X/22 to 10 192.32.X/22 and power it down

Sounds good -- I've moved ms-be2055's IP address from row D to row C in https://gerrit.wikimedia.org/r/c/operations/dns/+/539982 and powered it down, thanks!

Papaul added a comment.Oct 1 2019, 3:43 PM

disable switch port on asw-d-codfw for ms-be2055 since we are moving the server in row C

papaul@asw-d-codfw# show | compare 
[edit interfaces interface-range vlan-private1-d-codfw]
-    member xe-4/0/15;
[edit interfaces interface-range disabled]
     member ge-6/0/2 { ... }
+    member xe-4/0/15;
[edit interfaces]
-   xe-4/0/15 {
-       description ms-be2055;
-   }
Papaul added a comment.Oct 1 2019, 4:51 PM

switch port configuration for ms-be2055 in row C

apaul@asw-c-codfw# show | compare 
[edit interfaces interface-range vlan-private1-c-codfw]
     member xe-7/0/10 { ... }
+    member xe-2/0/13;
[edit interfaces]
+   xe-2/0/13 {
+       description ms-be2055;
+   }
Papaul reassigned this task from Papaul to fgiunchedi.Oct 1 2019, 4:53 PM

Change 540212 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: add ms-be205[1-6]

https://gerrit.wikimedia.org/r/540212

Change 540212 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: add ms-be205[1-6]

https://gerrit.wikimedia.org/r/540212

Change 540213 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/software/swift-ring@master] codfw-prod: add ms-be2051, minimal weight and servers_per_port

https://gerrit.wikimedia.org/r/540213

Change 540213 merged by Filippo Giunchedi:
[operations/software/swift-ring@master] codfw-prod: add ms-be2051, minimal weight and servers_per_port

https://gerrit.wikimedia.org/r/540213

Mentioned in SAL (#wikimedia-operations) [2019-10-01T21:34:40Z] <godog> swift codfw-prod: add ms-be2051 with minimal weight - T233638 T222366

Mentioned in SAL (#wikimedia-operations) [2019-10-02T15:29:56Z] <godog> swift codfw-prod: add ms-be2051 T233638

fgiunchedi moved this task from Backlog to Blocked on the User-fgiunchedi board.

Mentioned in SAL (#wikimedia-operations) [2019-10-07T21:20:13Z] <godog> swift codfw-prod: add ms-be205[3456] - T233638

Mentioned in SAL (#wikimedia-operations) [2019-10-08T18:53:15Z] <godog> codfw-prod: more weight to ms-be205[1-6] - T233638

fgiunchedi moved this task from Blocked to Radar on the User-fgiunchedi board.Oct 9 2019, 11:30 PM
fgiunchedi moved this task from Radar to Doing on the User-fgiunchedi board.

Mentioned in SAL (#wikimedia-operations) [2019-10-10T19:25:28Z] <godog> swift codfw-prod: more weight to ms-be205[1-6] - T233638

Mentioned in SAL (#wikimedia-operations) [2019-10-17T09:57:28Z] <godog> swift codfw-prod: more weight to ms-be205[1-6] - T233638

Mentioned in SAL (#wikimedia-operations) [2019-10-21T08:03:36Z] <godog> swift codfw-prod: final weight to ms-be205[1-6] - T233638

fgiunchedi closed this task as Resolved.Oct 22 2019, 4:40 PM

This is completed, hosts are fully in service now.