Page MenuHomePhabricator
Create Task
Maniphest T239249

(Need By: Jan 15) codfw: rack/setup/install mc-gp200[123].codfw.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and installation of 3 new memcache servers for codfw.

Shared Info

Hostnames: From mc-gp2001.codfw.wmnet to mc-gp2003.codfw.wmnet
Racking Proposal: If possible one host for each row, and not in racks where we have other mcXXXX hosts and In rack with 10G switch.
Partitioning/Raid: Same as the current mc20XX hosts.

Individual Server Checklists

mc-gp2001: row A rack A4/ xe-4/0/21

  • - receive in system on procurement task T238095
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1-row-vlan)
    • end on-site specific steps
  • - production dns entries added (private subnet for its row)
  • - operations/puppet update (install_server at minimum, other files if possible):
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

mc-gp2002: row B rack B4/ xe-4/0/6

  • - receive in system on procurement task T238095
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1-row-vlan)
    • end on-site specific steps
  • - production dns entries added (private subnet for its row)
  • - operations/puppet update (install_server at minimum, other files if possible):
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

mc-gp2003: row D rack D4/ xe-4/0/16

  • - receive in system on procurement task T238095
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, private1-row-vlan)
    • end on-site specific steps
  • - production dns entries added (private subnet for its row)
  • - operations/puppet update (install_server at minimum, other files if possible):
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Add role to mc-gp200[1-3]operations/puppetproduction+27 -0
role::memcached: move base::mysterious_sysctl into the perf profileoperations/puppetproduction+2 -1
role::memcached: move interface::rps settings to dedicated profileoperations/puppetproduction+12 -3
DHCP: Add MAC address entries for mc-gp200[1-3]operations/puppetproduction+18 -0
DNS:Add mgmt and production DNS for mc-gp200[1-3]operations/dnsmaster+18 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

Papaul created this task.Nov 26 2019, 5:14 PM
Restricted Application added a project: SRE. · View Herald TranscriptNov 26 2019, 5:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Papaul triaged this task as Medium priority.Nov 26 2019, 5:14 PM
Papaul updated the task description. (Show Details)Nov 27 2019, 4:35 PM
Papaul moved this task from Backlog to Racking Tasks on the ops-codfw board.Dec 2 2019, 4:11 PM
elukey added a comment.Dec 10 2019, 5:35 PM

@Papaul we might want to have a different naming convention for these hosts, please wait until I have a consensus before adding dns entries :)

Papaul updated the task description. (Show Details)Jan 2 2020, 6:19 PM
wiki_willy renamed this task from codfw: rack/setup/install mc203[7,8,9].codfw.wmnet to (No Need By Date Provided) codfw: rack/setup/install mc203[7,8,9].codfw.wmnet.Jan 2 2020, 11:43 PM
elukey renamed this task from (No Need By Date Provided) codfw: rack/setup/install mc203[7,8,9].codfw.wmnet to (Need By: Jan 15) codfw: rack/setup/install mc-gp200[123].codfw.wmnet.Jan 3 2020, 7:31 AM
elukey mentioned this in T241796: (Need By: Jan 15) rack/setup/install mc-gp200[123].codfw.wmnet.
elukey updated the task description. (Show Details)
Papaul updated the task description. (Show Details)Jan 6 2020, 6:24 PM
Papaul updated the task description. (Show Details)Jan 7 2020, 12:06 AM
Papaul updated the task description. (Show Details)Jan 7 2020, 12:17 AM
Papaul added a comment.Jan 7 2020, 1:11 AM
papaul@asw-a-codfw> show interfaces xe-4/0/21 descriptions 
Interface       Admin Link Description
xe-4/0/21       up    up   mc-gp2001
papaul@asw-b-codfw# run show interfaces xe-4/0/6 descriptions 
Interface       Admin Link Description
xe-4/0/6        up    up   mc-gp2002
papaul@asw-d-codfw# run show interfaces xe-4/0/16 descriptions 
Interface       Admin Link Description
xe-4/0/16       up    up   mc-gp2003
Papaul updated the task description. (Show Details)Jan 7 2020, 1:12 AM
Papaul updated the task description. (Show Details)Jan 8 2020, 9:37 PM
gerritbot added a comment.Jan 9 2020, 1:19 AM

Change 563004 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS:Add mgmt and production DNS for mc-gp200[1-3]

https://gerrit.wikimedia.org/r/563004

gerritbot added a project: Patch-For-Review.Jan 9 2020, 1:19 AM
Papaul added a comment.Jan 9 2020, 1:30 AM

@elukey are we installing Buster or Stretch ?

gerritbot added a comment.Jan 9 2020, 2:43 AM

Change 563004 merged by Papaul:
[operations/dns@master] DNS:Add mgmt and production DNS for mc-gp200[1-3]

https://gerrit.wikimedia.org/r/563004

Papaul updated the task description. (Show Details)Jan 9 2020, 2:45 AM
Maintenance_bot removed a project: Patch-For-Review.Jan 9 2020, 3:10 AM
elukey added a comment.Jan 9 2020, 7:23 AM
In T239249#5788574, @Papaul wrote:

@elukey are we installing Buster or Stretch ?

Buster please! :)

jijiki added a parent task: T240684: Test gutter pool failover in production and memcached 1.5.x.Jan 9 2020, 12:26 PM
gerritbot added a comment.Jan 9 2020, 2:54 PM

Change 563184 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP: Add MAC address entries for mc-gp200[1-3]

https://gerrit.wikimedia.org/r/563184

gerritbot added a project: Patch-For-Review.Jan 9 2020, 2:54 PM
gerritbot added a comment.Jan 9 2020, 7:09 PM

Change 563184 merged by Elukey:
[operations/puppet@production] DHCP: Add MAC address entries for mc-gp200[1-3]

https://gerrit.wikimedia.org/r/563184

Maintenance_bot removed a project: Patch-For-Review.Jan 9 2020, 7:10 PM
Papaul updated the task description. (Show Details)Jan 9 2020, 7:40 PM
Papaul updated the task description. (Show Details)Jan 9 2020, 7:50 PM
Papaul updated the task description. (Show Details)Jan 9 2020, 8:06 PM
Papaul reassigned this task from Papaul to elukey.Jan 9 2020, 8:35 PM
Papaul updated the task description. (Show Details)

@elukey all yours

gerritbot added a comment.Jan 10 2020, 8:51 AM

Change 563381 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add role to mc-gp200[1-3]

https://gerrit.wikimedia.org/r/563381

gerritbot added a project: Patch-For-Review.Jan 10 2020, 8:51 AM
gerritbot added a comment.Jan 13 2020, 9:40 AM

Change 563969 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::memcached: move interface::rps settings to dedicated profile

https://gerrit.wikimedia.org/r/563969

gerritbot added a comment.Jan 13 2020, 10:03 AM

Change 563969 merged by Elukey:
[operations/puppet@production] role::memcached: move interface::rps settings to dedicated profile

https://gerrit.wikimedia.org/r/563969

gerritbot added a comment.Jan 13 2020, 10:22 AM

Change 563979 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::memcached: move base::mysterious_sysctl into the perf profile

https://gerrit.wikimedia.org/r/563979

gerritbot added a comment.Jan 13 2020, 10:26 AM

Change 563979 merged by Elukey:
[operations/puppet@production] role::memcached: move base::mysterious_sysctl into the perf profile

https://gerrit.wikimedia.org/r/563979

gerritbot added a comment.Jan 13 2020, 10:34 AM

Change 563381 merged by Elukey:
[operations/puppet@production] Add role to mc-gp200[1-3]

https://gerrit.wikimedia.org/r/563381

Maintenance_bot removed a project: Patch-For-Review.Jan 13 2020, 11:11 AM
elukey closed this task as Resolved.Jan 13 2020, 11:17 AM
elukey updated the task description. (Show Details)
jijiki added a parent task: T244852: Upgrade and improve our application object caching service (memcached).Feb 11 2020, 12:54 PM
jijiki removed a parent task: T240684: Test gutter pool failover in production and memcached 1.5.x.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits