Page MenuHomePhabricator
Create Task
Maniphest T187447

Decommission restbase-test200[123]
Closed, ResolvedPublic
Actions

Description

See the parent task (for Services to confirm offline) for step 1.

These three hosts were purchased back in 2013-01-12, and are over 5 years old. So they will be decommissioned, not reclaimed to spares.

restbase-test2001:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal) asw-b-codfw:ge-5/0/19: restbase-test2001
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite) - please note as ssds these need the smartctl utility wipe, NOT a normal HDD wipe.
  • - system unracked and decommissioned (by onsite), update racktables with result
  • - switch port configration removed from switch once system is unracked.
  • - add system to decommission tracking google sheet
  • - mgmt dns entries removed.

restbase-test2002:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal) asw-b-codfw:ge-5/0/18: restbase-test2002
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite) - please note as ssds these need the smartctl utility wipe, NOT a normal HDD wipe.
  • - system unracked and decommissioned (by onsite), update racktables with result
  • - switch port configration removed from switch once system is unracked.
  • - add system to decommission tracking google sheet
  • - mgmt dns entries removed.

restbase-test2003:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal) asw-b-codfw:ge-5/0/19: restbase-test2001
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite) - please note as ssds these need the smartctl utility wipe, NOT a normal HDD wipe.
  • - system unracked and decommissioned (by onsite), update racktables with result
  • - switch port configration removed from switch once system is unracked.
  • - add system to decommission tracking google sheet
  • - mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
DNS: Remove DNS entries for restbase-test200[1-3]operations/dnsmaster+2 -12
restbase-test200* prod dns removaloperations/dnsmaster+0 -18
decom restbase-test200[123]operations/puppetproduction+0 -96
Customize query in gerrit

Related Objects
Search...

Event Timeline

faidon triaged this task as Low priority.Feb 15 2018, 2:00 PM
faidon created this task.
Eevans added a comment.Feb 20 2018, 5:57 PM

Confirmed in the Services team meeting today; These machines can be decommissioned at the earliest convenience!

RobH added a project: hardware-requests.Feb 20 2018, 6:28 PM
RobH subscribed.

Please ensure all decom requests are tagged with #hw-requests.

RobH updated the task description. (Show Details)Feb 20 2018, 7:07 PM
RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.Mar 5 2018, 7:54 PM
fgiunchedi updated the task description. (Show Details)Mar 12 2018, 2:55 PM
RobH claimed this task.Mar 13 2018, 9:20 PM
RobH updated the task description. (Show Details)
RobH updated the task description. (Show Details)Mar 13 2018, 9:29 PM
gerritbot subscribed.Mar 13 2018, 9:33 PM

Change 419312 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom restbase-test200[123]

https://gerrit.wikimedia.org/r/419312

gerritbot added a project: Patch-For-Review.Mar 13 2018, 9:33 PM

Change 419312 merged by RobH:
[operations/puppet@production] decom restbase-test200[123]

https://gerrit.wikimedia.org/r/419312

gerritbot added a comment.Mar 13 2018, 9:40 PM

Change 419315 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] restbase-test200* prod dns removal

https://gerrit.wikimedia.org/r/419315

gerritbot added a comment.Mar 13 2018, 9:40 PM

Change 419315 merged by RobH:
[operations/dns@master] restbase-test200* prod dns removal

https://gerrit.wikimedia.org/r/419315

RobH reassigned this task from RobH to Papaul.Mar 13 2018, 9:43 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)

These are ready to be wiped by onsite. Please note as SSDs, these need the specific smartctl utility run to erase them securely. The HDD method of writing zeros will NOT work.

RobH moved this task from Backlog to Decommission on the ops-codfw board.Mar 15 2018, 5:13 PM
Papaul raised the priority of this task from Low to Medium.Mar 26 2018, 8:37 PM
Papaul added a comment.Mar 28 2018, 4:13 PM

switch port information
asw-b5-codfw
restbase-test2001 ge-5/0/19
restbase-test2002 ge-5/0/16
restbase-test2003 ge-5/0/20

Papaul updated the task description. (Show Details)Mar 29 2018, 4:12 PM
Papaul reassigned this task from Papaul to RobH.Mar 29 2018, 4:36 PM
Papaul subscribed.

@RobH can you please work on the switch ports and assign back to me for mgmt DNS removal.

Thanks.

RobH reassigned this task from RobH to Papaul.EditedMar 29 2018, 4:43 PM

@Papaul:

The switch port info you provided does not match the switch's configuration:

In T187447#4088465, @Papaul wrote:

switch port information
asw-b5-codfw
restbase-test2001 ge-5/0/19
restbase-test2002 ge-5/0/16
restbase-test2003 ge-5/0/20

robh@asw-b-codfw> show interfaces descriptions | grep ge-5/0/19 
ge-5/0/19       down  down restbase-test2001

{master:2}
robh@asw-b-codfw> show interfaces descriptions | grep ge-5/0/16    
ge-5/0/16       down  down restbase-test2003

{master:2}
robh@asw-b-codfw> show interfaces descriptions | grep ge-5/0/20    
ge-5/0/20       up    up   labtestcontrol2001-eth0
ge-5/0/16       down  down restbase-test2003
ge-5/0/18       down  down restbase-test2002
ge-5/0/19       down  down restbase-test2001

Did you trace those out by hand and perhaps just have some mistakes? Can you re-check before I go disabling and removing ports please? It seems the switch is right, since the ports are down, and I'm guessing your update had the mistakes?

Please advise,

Papaul reassigned this task from Papaul to RobH.Apr 2 2018, 3:21 PM

Switch port confirmation
asw-b5-codfw
restbase-test2001 ge-5/0/19
restbase-test2002 ge-5/0/16
restbase-test2003 ge-5/0/18

RobH reassigned this task from RobH to Papaul.Apr 9 2018, 10:15 PM

I've removed the descriptions from those switch ports.

@Papaul: moving forward, you should be aware that the order of operation of removing the switch port description can happen AFTER removing the mgmt dns entries. The main thing is that neither one of those should happen until after you have removed it from the rack.

Hope that clarifies!

RobH updated the task description. (Show Details)Apr 9 2018, 10:15 PM
gerritbot added a comment.Apr 12 2018, 3:53 PM

Change 425842 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove DNS entries for restbase-test200[1-3]

https://gerrit.wikimedia.org/r/425842

gerritbot added a project: Patch-For-Review.Apr 12 2018, 3:54 PM
Papaul updated the task description. (Show Details)Apr 12 2018, 3:57 PM
gerritbot added a comment.Apr 12 2018, 9:04 PM

Change 425842 merged by Dzahn:
[operations/dns@master] DNS: Remove DNS entries for restbase-test200[1-3]

https://gerrit.wikimedia.org/r/425842

Papaul closed this task as Resolved.Apr 12 2018, 9:22 PM
Papaul updated the task description. (Show Details)
Papaul updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL