Page MenuHomePhabricator
Create Task
Maniphest T221817

Decommission labcontrol1001 & labcontrol1002
Closed, ResolvedPublic
Actions

Description

labcontrol1001:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.)

Steps for DC-Ops:
The following steps cannot be interrupted, as it will leave the system in an unfinished state.
Start non-interrupt steps:

  • - disable puppet on host
  • - power down host
  • - update netbox status to Inventory (if decom) or Planned (if spare)
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate (handled by wmf-decommission-host)
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key (handled by wmf-decommission-host)

End non-interrupt steps.

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.
  • - change netbox status to offline when unracked

labcontrol1002:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.)

Steps for DC-Ops:
The following steps cannot be interrupted, as it will leave the system in an unfinished state.
Start non-interrupt steps:

  • - disable puppet on host
  • - power down host
  • - update netbox status to Inventory (if decom) or Planned (if spare)
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate (handled by wmf-decommission-host)
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key (handled by wmf-decommission-host)

End non-interrupt steps.

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.
  • - change netbox status to offline when unracked

Details

SubjectRepoBranchLines +/-
DNS: Remove mgmt DNS for labcontrol100[1-2]operations/dnsmaster+1 -9
decom labcontrol100[12]operations/puppetproduction+0 -8
labcontrol100[12] production dns decomoperations/dnsmaster+1 -9
Move labcontrol1001/1002 to role::spare and clean up referencesoperations/puppetproduction+5 -30
Customize query in gerrit

Related Objects

Event Timeline

Andrew created this task.Apr 25 2019, 4:24 AM
gerritbot added a comment.Apr 25 2019, 4:41 AM

Change 506345 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Move labcontrol1001/1002 to role::spare and clean up references

https://gerrit.wikimedia.org/r/506345

gerritbot added a project: Patch-For-Review.Apr 25 2019, 4:41 AM
Andrew renamed this task from Reclaim/Decommission labcontrol1001, 1002 to Decommission labcontrol1001, 1002.Apr 25 2019, 4:27 PM
gerritbot added a comment.Apr 25 2019, 9:21 PM

Change 506345 merged by Andrew Bogott:
[operations/puppet@production] Move labcontrol1001/1002 to role::spare and clean up references

https://gerrit.wikimedia.org/r/506345

Andrew reassigned this task from Andrew to RobH.Apr 25 2019, 10:47 PM
Andrew updated the task description. (Show Details)
Andrew added a subscriber: RobH.

@RobH, I'm supposed to assign decom hosts to you at this point, right?

RobH removed a project: Patch-For-Review.Apr 25 2019, 10:52 PM
RobH updated the task description. (Show Details)

labcontrol1001:asw2-c-eqiad:ge-5/0/32
labcontrol1002:asw2-c-eqiad:ge-7/0/26

RobH added a comment.Apr 25 2019, 10:55 PM
In T221817#5139201, @Andrew wrote:

@RobH, I'm supposed to assign decom hosts to you at this point, right?

Yep! Basically once we get to the point of disabling puppet and powering down a host, it also has to immediately have the switch port disabled. So only folks who can change switch config should do those steps.

ops-monitoring-bot added a comment.Apr 25 2019, 10:56 PM

cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: labcontrol1001.wikimedia.org

  • labcontrol1001.wikimedia.org
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
ops-monitoring-bot added a comment.Apr 25 2019, 10:57 PM

cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: labcontrol1002.wikimedia.org

  • labcontrol1002.wikimedia.org
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
RobH triaged this task as Medium priority.Apr 25 2019, 10:59 PM
RobH added a project: ops-eqiad.
RobH updated the task description. (Show Details)
RobH moved this task from Backlog to Decommission on the ops-eqiad board.
RobH moved this task from Backlog to Ready for Decommission on the decommission-hardware board.
gerritbot added a comment.Apr 25 2019, 11:01 PM

Change 506567 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] labcontrol100[12] production dns decom

https://gerrit.wikimedia.org/r/506567

gerritbot added a project: Patch-For-Review.Apr 25 2019, 11:01 PM

Change 506567 merged by RobH:
[operations/dns@master] labcontrol100[12] production dns decom

https://gerrit.wikimedia.org/r/506567

gerritbot added a comment.Apr 25 2019, 11:04 PM

Change 506569 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom labcontrol100[12]

https://gerrit.wikimedia.org/r/506569

gerritbot added a comment.Apr 25 2019, 11:23 PM

Change 506569 merged by RobH:
[operations/puppet@production] decom labcontrol100[12]

https://gerrit.wikimedia.org/r/506569

RobH reassigned this task from RobH to • Cmjohnson.Apr 25 2019, 11:26 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH moved this task from Ready for Decommission to pending onsite steps (eqiad) on the decommission-hardware board.
RobH renamed this task from Decommission labcontrol1001, 1002 to Decommission labcontrol1001 & labcontrol1002.Apr 25 2019, 11:49 PM
RobH updated the task description. (Show Details)Apr 25 2019, 11:51 PM
MoritzMuehlenhoff mentioned this in T221911: Degraded RAID on labcontrol1001.Apr 26 2019, 6:59 AM
MoritzMuehlenhoff mentioned this in T221910: Degraded RAID on labcontrol1001.
MoritzMuehlenhoff mentioned this in T221909: Degraded RAID on labcontrol1002.
MoritzMuehlenhoff mentioned this in T221912: Degraded RAID on labcontrol1002.
Andrew mentioned this in T212772: Track remaining trusty servers in production.Apr 26 2019, 12:43 PM
• Cmjohnson reassigned this task from • Cmjohnson to Jclark-ctr.Sep 19 2019, 8:55 PM
• Cmjohnson subscribed.

John, please wipe the servers, remove from the rack, update netbox and the tracking sheet. Assign back to me once you finish so I can kill the switch ports.

Jclark-ctr updated the task description. (Show Details)Oct 11 2019, 10:43 PM
Papaul subscribed.Oct 17 2019, 11:25 PM
papaul@asw2-c-eqiad# show | compare 
[edit interfaces]
-   ge-5/0/32 {
-       description labcontrol1001;
-   }
-   ge-7/0/26 {
-       description labcontrol1002;
-   }
Papaul updated the task description. (Show Details)Oct 17 2019, 11:26 PM
gerritbot added a comment.Oct 17 2019, 11:30 PM

Change 544075 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for labcontrol100[1-2]

https://gerrit.wikimedia.org/r/544075

gerritbot added a project: Patch-For-Review.Oct 17 2019, 11:30 PM

Change 544075 merged by Papaul:
[operations/dns@master] DNS: Remove mgmt DNS for labcontrol100[1-2]

https://gerrit.wikimedia.org/r/544075

Papaul closed this task as Resolved.Oct 17 2019, 11:33 PM
Papaul updated the task description. (Show Details)

Complete

Maintenance_bot removed a project: Patch-For-Review.Oct 18 2019, 12:10 AM
RobH unsubscribed.Oct 18 2019, 5:53 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL