Page MenuHomePhabricator
Create Task
Maniphest T233638

rack/setup/install ms-be205[1-6].codfw.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking and setup of 6 new ms-be systems in codfw.

This is expanding the backend service, not replacing any existing servers.

Hostnames: ms-be205[1-6].codfw.wmnet
Racking Proposal: 10G racks, 2x hosts in rows A+C. 1x host in rows B+D. Rack diversity with existing ms-be hw is preferred but not required.
Networking/Subnet/VLAN/IP: Internal subnet per row
Partitioning/Raid: use existing ms-be setup

ms-be2051:

  • - receive in system on procurement task T228460
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be2052:

  • - receive in system on procurement task T228460
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be2053:

  • - receive in system on procurement task T228460
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be2054:

  • - receive in system on procurement task T228460
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be2055:

  • - receive in system on procurement task T228460
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

ms-be2056:

  • - receive in system on procurement task T228460
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
codfw-prod: add ms-be2051, minimal weight and servers_per_portoperations/software/swift-ringmaster+1 K -1 K
hieradata: add ms-be205[1-6]operations/puppetproduction+6 -0
Move ms-be2055 to row Coperations/dnsmaster+2 -2
DHCP: Add MAC address entries for ms-be205[1-6]operations/puppetproduction+30 -0
DNS: Add mgmt and production DNS for ms-be205[1-6]operations/dnsmaster+36 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedRobHT227314 eqiad+codfw: 6x hardware request for swift backend (each site)
Unknown Object (Task)
 ResolvedfgiunchediT233638 rack/setup/install ms-be205[1-6].codfw.wmnet

Event Timeline

RobH created this task.Sep 23 2019, 4:25 PM
RobH triaged this task as Medium priority.
Restricted Application added a project: SRE. · View Herald TranscriptSep 23 2019, 4:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
RobH added a parent task: Unknown Object (Task).Sep 23 2019, 4:25 PM
Papaul updated the task description. (Show Details)Sep 24 2019, 4:03 PM
Papaul updated the task description. (Show Details)Sep 24 2019, 9:31 PM
Papaul updated the task description. (Show Details)Sep 25 2019, 2:01 PM
Papaul updated the task description. (Show Details)Sep 25 2019, 2:25 PM
Papaul moved this task from Backlog to Racking Tasks on the ops-codfw board.
Papaul updated the task description. (Show Details)Sep 26 2019, 3:50 PM
gerritbot added a comment.Sep 26 2019, 8:17 PM

Change 539412 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Add mgmt and production DNS for ms-be205[1-6]

https://gerrit.wikimedia.org/r/539412

gerritbot added a project: Patch-For-Review.Sep 26 2019, 8:17 PM
gerritbot added a comment.Sep 26 2019, 8:32 PM

Change 539412 merged by Papaul:
[operations/dns@master] DNS: Add mgmt and production DNS for ms-be205[1-6]

https://gerrit.wikimedia.org/r/539412

Papaul updated the task description. (Show Details)Sep 26 2019, 8:40 PM
Papaul updated the task description. (Show Details)Sep 26 2019, 9:00 PM
Maintenance_bot removed a project: Patch-For-Review.Sep 26 2019, 9:10 PM
gerritbot added a comment.Sep 26 2019, 10:55 PM

Change 539432 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP: Add MAC address entries for ms-be205[1-6]

https://gerrit.wikimedia.org/r/539432

gerritbot added a project: Patch-For-Review.Sep 26 2019, 10:55 PM
gerritbot added a comment.Sep 26 2019, 11:05 PM

Change 539432 merged by Papaul:
[operations/puppet@production] DHCP: Add MAC address entries for ms-be205[1-6]

https://gerrit.wikimedia.org/r/539432

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2019, 11:10 PM
Papaul updated the task description. (Show Details)Sep 27 2019, 2:08 PM
Papaul updated the task description. (Show Details)Sep 27 2019, 3:06 PM
Papaul updated the task description. (Show Details)Sep 27 2019, 4:06 PM
Papaul updated the task description. (Show Details)Sep 27 2019, 5:06 PM
Papaul updated the task description. (Show Details)Sep 27 2019, 5:46 PM
RobH unsubscribed.Sep 27 2019, 6:00 PM
Papaul updated the task description. (Show Details)Sep 27 2019, 6:04 PM
Papaul reassigned this task from Papaul to fgiunchedi.Sep 27 2019, 10:32 PM
Papaul updated the task description. (Show Details)
Papaul subscribed.

@fgiunchedi all yours

fgiunchedi reassigned this task from fgiunchedi to Papaul.Sep 30 2019, 1:28 PM
In T233638#5531188, @Papaul wrote:

@fgiunchedi all yours

Thanks @Papaul !

I see two hosts in row A+D per netbox, and one in B+C, although it should be two hosts in A+C and one in B+D. In other words there's a misplace host in row D that should be in row C instead.

Papaul added a comment.Sep 30 2019, 2:16 PM

@sorry I fogot to mentioned that on the task, It was the faster and easier way to rack those servers. We can still move 1 host from D to C but it will take a while for me to get in into C2 if you find with that.

If you want to move 1 host from D to C please change the IP address of ms-be2055 from 10.192.48.X/22 to 10 192.32.X/22 and power it down

Thanks.

gerritbot added a comment.Sep 30 2019, 10:12 PM

Change 539982 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/dns@master] Move ms-be2055 to row C

https://gerrit.wikimedia.org/r/539982

gerritbot added a project: Patch-For-Review.Sep 30 2019, 10:12 PM

Change 539982 merged by Filippo Giunchedi:
[operations/dns@master] Move ms-be2055 to row C

https://gerrit.wikimedia.org/r/539982

fgiunchedi added a comment.Sep 30 2019, 10:16 PM
In T233638#5534157, @Papaul wrote:

@sorry I fogot to mentioned that on the task, It was the faster and easier way to rack those servers. We can still move 1 host from D to C but it will take a while for me to get in into C2 if you find with that.

If you want to move 1 host from D to C please change the IP address of ms-be2055 from 10.192.48.X/22 to 10 192.32.X/22 and power it down

Sounds good -- I've moved ms-be2055's IP address from row D to row C in https://gerrit.wikimedia.org/r/c/operations/dns/+/539982 and powered it down, thanks!

Maintenance_bot removed a project: Patch-For-Review.Sep 30 2019, 11:10 PM
Papaul added a comment.Oct 1 2019, 3:43 PM

disable switch port on asw-d-codfw for ms-be2055 since we are moving the server in row C

papaul@asw-d-codfw# show | compare 
[edit interfaces interface-range vlan-private1-d-codfw]
-    member xe-4/0/15;
[edit interfaces interface-range disabled]
     member ge-6/0/2 { ... }
+    member xe-4/0/15;
[edit interfaces]
-   xe-4/0/15 {
-       description ms-be2055;
-   }
Papaul added a comment.Oct 1 2019, 4:51 PM

switch port configuration for ms-be2055 in row C

apaul@asw-c-codfw# show | compare 
[edit interfaces interface-range vlan-private1-c-codfw]
     member xe-7/0/10 { ... }
+    member xe-2/0/13;
[edit interfaces]
+   xe-2/0/13 {
+       description ms-be2055;
+   }
Papaul reassigned this task from Papaul to fgiunchedi.Oct 1 2019, 4:53 PM
gerritbot added a comment.Oct 1 2019, 9:01 PM

Change 540212 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: add ms-be205[1-6]

https://gerrit.wikimedia.org/r/540212

gerritbot added a project: Patch-For-Review.Oct 1 2019, 9:01 PM
gerritbot added a comment.Oct 1 2019, 9:06 PM

Change 540212 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: add ms-be205[1-6]

https://gerrit.wikimedia.org/r/540212

Maintenance_bot removed a project: Patch-For-Review.Oct 1 2019, 9:10 PM
gerritbot added a comment.Oct 1 2019, 9:14 PM

Change 540213 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/software/swift-ring@master] codfw-prod: add ms-be2051, minimal weight and servers_per_port

https://gerrit.wikimedia.org/r/540213

gerritbot added a project: Patch-For-Review.Oct 1 2019, 9:14 PM
gerritbot added a comment.Oct 1 2019, 9:26 PM

Change 540213 merged by Filippo Giunchedi:
[operations/software/swift-ring@master] codfw-prod: add ms-be2051, minimal weight and servers_per_port

https://gerrit.wikimedia.org/r/540213

Stashbot added a comment.Oct 1 2019, 9:34 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-01T21:34:40Z] <godog> swift codfw-prod: add ms-be2051 with minimal weight - T233638 T222366

Stashbot mentioned this in T222366: Test swift object server deployment with one disk per tcp port.Oct 1 2019, 9:34 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 1 2019, 10:10 PM
Stashbot added a comment.Oct 2 2019, 3:29 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-02T15:29:56Z] <godog> swift codfw-prod: add ms-be2051 T233638

fgiunchedi added a project: User-fgiunchedi.Oct 7 2019, 9:14 PM
fgiunchedi moved this task from Backlog to Other on the User-fgiunchedi board.
Stashbot added a comment.Oct 7 2019, 9:20 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-07T21:20:13Z] <godog> swift codfw-prod: add ms-be205[3456] - T233638

Stashbot added a comment.Oct 8 2019, 6:53 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-08T18:53:15Z] <godog> codfw-prod: more weight to ms-be205[1-6] - T233638

fgiunchedi moved this task from Other to Radar on the User-fgiunchedi board.Oct 9 2019, 11:30 PM
fgiunchedi moved this task from Radar to Doing on the User-fgiunchedi board.
Stashbot added a comment.Oct 10 2019, 7:25 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-10T19:25:28Z] <godog> swift codfw-prod: more weight to ms-be205[1-6] - T233638

Stashbot added a comment.Oct 17 2019, 9:57 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-17T09:57:28Z] <godog> swift codfw-prod: more weight to ms-be205[1-6] - T233638

Stashbot added a comment.Oct 21 2019, 8:03 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-21T08:03:36Z] <godog> swift codfw-prod: final weight to ms-be205[1-6] - T233638

fgiunchedi closed this task as Resolved.Oct 22 2019, 4:40 PM

This is completed, hosts are fully in service now.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits