Page MenuHomePhabricator

Decommission mw2017
Closed, ResolvedPublic

Description

These two servers (mw2017 & mw2099) are the codfw test servers, and have be converted to VMs. mw2099 seems to have been decommissioned via another task already, so this just covers mw2017.

mw2017:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal) asw-a-cofw:ge-3/0/16
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

Related Gerrit Patches:
operations/dns : masterDNS: Remove mgmt DNS for mw2017
operations/dns : masterdecom mw2017 prod dns entries
operations/puppet : productiondecom mw2017 and mw2099
operations/puppet : productioncodfw: decommission mw2017-2099

Related Objects

Event Timeline

Joe created this task.Feb 15 2018, 4:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 15 2018, 4:52 PM
fgiunchedi triaged this task as Medium priority.Feb 16 2018, 9:41 AM
Cmjohnson assigned this task to Papaul.Feb 16 2018, 3:47 PM
Cmjohnson edited projects, added ops-codfw; removed ops-eqiad.
Papaul reassigned this task from Papaul to RobH.Feb 16 2018, 9:01 PM
Papaul added a subscriber: Papaul.

Change 416968 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] codfw: decommission mw2017-2099

https://gerrit.wikimedia.org/r/416968

Change 416968 merged by Giuseppe Lavagetto:
[operations/puppet@production] codfw: decommission mw2017-2099

https://gerrit.wikimedia.org/r/416968

Mentioned in SAL (#wikimedia-operations) [2018-03-08T09:56:31Z] <_joe_> decommissioning mw2017-2099 T187467

Joe updated the task description. (Show Details)Mar 8 2018, 10:02 AM
RobH moved this task from Backlog to Decommission on the ops-codfw board.Mar 15 2018, 5:13 PM
RobH renamed this task from Decommission mw2017 and mw2099 to Decommission mw2017 & mw2099.Jul 23 2018, 5:15 PM
RobH updated the task description. (Show Details)

Change 447476 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom mw2017 and mw2099

https://gerrit.wikimedia.org/r/447476

RobH added a comment.Jul 23 2018, 5:21 PM

mw2099 seems to have been decommissioned and removed via another task, since its fully out and in the decom rack. I'm removing it from this task.

RobH renamed this task from Decommission mw2017 & mw2099 to Decommission mw2017.Jul 23 2018, 5:22 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH updated the task description. (Show Details)Jul 23 2018, 5:43 PM

Change 447476 merged by RobH:
[operations/puppet@production] decom mw2017 and mw2099

https://gerrit.wikimedia.org/r/447476

Change 447481 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom mw2017 prod dns entries

https://gerrit.wikimedia.org/r/447481

Change 447481 merged by RobH:
[operations/dns@master] decom mw2017 prod dns entries

https://gerrit.wikimedia.org/r/447481

RobH reassigned this task from RobH to Papaul.Jul 23 2018, 5:50 PM
RobH moved this task from Backlog to pending onsite steps (codfw) on the decommission board.
RobH added a subscriber: RobH.
Papaul added a comment.Aug 8 2018, 3:26 PM

Disk wipe in progress

Papaul updated the task description. (Show Details)Aug 8 2018, 3:26 PM
Papaul updated the task description. (Show Details)Aug 9 2018, 4:53 PM
Papaul added a comment.Aug 9 2018, 5:12 PM

switch port information
ge-3/0/16

show interfaces ge-3/0/16 
Physical interface: ge-3/0/16, Administratively down, Physical link is Down
  Interface index: 1187, SNMP ifIndex: 557
  Description: DISABLED
Papaul updated the task description. (Show Details)Aug 9 2018, 5:12 PM

Change 451668 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for mw2017

https://gerrit.wikimedia.org/r/451668

Change 451668 merged by Dzahn:
[operations/dns@master] DNS: Remove mgmt DNS for mw2017

https://gerrit.wikimedia.org/r/451668

Dzahn updated the task description. (Show Details)Aug 9 2018, 5:26 PM
Papaul closed this task as Resolved.Aug 9 2018, 7:34 PM