Page MenuHomePhabricator
Create Task
Maniphest T187467

Decommission mw2017
Closed, ResolvedPublic
Actions

Description

These two servers (mw2017 & mw2099) are the codfw test servers, and have be converted to VMs. mw2099 seems to have been decommissioned via another task already, so this just covers mw2017.

mw2017:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal) asw-a-cofw:ge-3/0/16
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
DNS: Remove mgmt DNS for mw2017operations/dnsmaster+1 -2
decom mw2017 prod dns entriesoperations/dnsmaster+1 -2
decom mw2017 and mw2099operations/puppetproduction+0 -12
codfw: decommission mw2017-2099operations/puppetproduction+6 -6
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedPapaulT187467 Decommission mw2017
 ResolvedJoeT187468 Create 2 VMs in codfw for mwdebug20001 and 2002

Event Timeline

Joe created this task.Feb 15 2018, 4:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 15 2018, 4:52 PM
Joe added projects: SRE, ops-eqiad.Feb 15 2018, 4:54 PM
fgiunchedi triaged this task as Medium priority.Feb 16 2018, 9:41 AM
• Cmjohnson assigned this task to Papaul.Feb 16 2018, 3:47 PM
• Cmjohnson edited projects, added ops-codfw; removed ops-eqiad.
Papaul reassigned this task from Papaul to RobH.Feb 16 2018, 9:01 PM
Papaul subscribed.
gerritbot subscribed.Mar 7 2018, 3:58 PM

Change 416968 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] codfw: decommission mw2017-2099

https://gerrit.wikimedia.org/r/416968

gerritbot added a project: Patch-For-Review.Mar 7 2018, 3:58 PM
Joe closed subtask T187468: Create 2 VMs in codfw for mwdebug20001 and 2002 as Resolved.Mar 8 2018, 9:21 AM
gerritbot added a comment.Mar 8 2018, 9:52 AM

Change 416968 merged by Giuseppe Lavagetto:
[operations/puppet@production] codfw: decommission mw2017-2099

https://gerrit.wikimedia.org/r/416968

Stashbot subscribed.Mar 8 2018, 9:56 AM

Mentioned in SAL (#wikimedia-operations) [2018-03-08T09:56:31Z] <_joe_> decommissioning mw2017-2099 T187467

Joe updated the task description. (Show Details)Mar 8 2018, 10:02 AM
RobH moved this task from Backlog to Decommission on the ops-codfw board.Mar 15 2018, 5:13 PM
MoritzMuehlenhoff added a project: hardware-requests.Apr 10 2018, 7:39 AM
RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.Apr 11 2018, 5:46 PM
RobH edited projects, added decommission-hardware; removed hardware-requests, Patch-For-Review.May 3 2018, 10:57 PM
RobH renamed this task from Decommission mw2017 and mw2099 to Decommission mw2017 & mw2099.Jul 23 2018, 5:15 PM
RobH updated the task description. (Show Details)
gerritbot added a comment.Jul 23 2018, 5:20 PM

Change 447476 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom mw2017 and mw2099

https://gerrit.wikimedia.org/r/447476

gerritbot added a project: Patch-For-Review.Jul 23 2018, 5:20 PM
RobH added a comment.Jul 23 2018, 5:21 PM

mw2099 seems to have been decommissioned and removed via another task, since its fully out and in the decom rack. I'm removing it from this task.

RobH renamed this task from Decommission mw2017 & mw2099 to Decommission mw2017.Jul 23 2018, 5:22 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH updated the task description. (Show Details)Jul 23 2018, 5:43 PM
gerritbot added a comment.Jul 23 2018, 5:44 PM

Change 447476 merged by RobH:
[operations/puppet@production] decom mw2017 and mw2099

https://gerrit.wikimedia.org/r/447476

gerritbot added a comment.Jul 23 2018, 5:48 PM

Change 447481 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom mw2017 prod dns entries

https://gerrit.wikimedia.org/r/447481

gerritbot added a project: Patch-For-Review.Jul 23 2018, 5:48 PM

Change 447481 merged by RobH:
[operations/dns@master] decom mw2017 prod dns entries

https://gerrit.wikimedia.org/r/447481

RobH reassigned this task from RobH to Papaul.Jul 23 2018, 5:50 PM
RobH moved this task from Backlog to pending onsite steps (codfw) on the decommission-hardware board.
RobH subscribed.
Papaul added a comment.Aug 8 2018, 3:26 PM

Disk wipe in progress

Papaul updated the task description. (Show Details)Aug 8 2018, 3:26 PM
Papaul updated the task description. (Show Details)Aug 9 2018, 4:53 PM
Papaul added a comment.Aug 9 2018, 5:12 PM

switch port information
ge-3/0/16

show interfaces ge-3/0/16 
Physical interface: ge-3/0/16, Administratively down, Physical link is Down
  Interface index: 1187, SNMP ifIndex: 557
  Description: DISABLED
Papaul updated the task description. (Show Details)Aug 9 2018, 5:12 PM
gerritbot added a comment.Aug 9 2018, 5:20 PM

Change 451668 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for mw2017

https://gerrit.wikimedia.org/r/451668

gerritbot added a comment.Aug 9 2018, 5:25 PM

Change 451668 merged by Dzahn:
[operations/dns@master] DNS: Remove mgmt DNS for mw2017

https://gerrit.wikimedia.org/r/451668

Dzahn updated the task description. (Show Details)Aug 9 2018, 5:26 PM
Papaul closed this task as Resolved.Aug 9 2018, 7:34 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits