Page MenuHomePhabricator
Create Task
Maniphest T226715

decommission restbase10(0[7-9]|1[0-5])
Closed, ResolvedPublicRequest
Actions

Description

Filing the actual decom task for these samsung ssd equipped servers, see also T223976: Decommission restbase10(0[7-9]|1[0-5]) and T208087: Replace remaining Samsung SSDs

restbase1007.eqiad.wmnet

The first 5 steps should be completed by the service owner that is returning the server to DC-ops (for reclaim to spare or decommissioning, dependent on server configuration and age.)

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system)
  • - unassign service owner from this task, check off completed steps, and assign to @RobH for followup on below steps.

Steps for DC-Ops:

The following steps cannot be interrupted, as it will leave the system in an unfinished state.

Start non-interrupt steps:

End non-interrupt steps.

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and change status of hardware to 'offline' when unracked.
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

restbase1008.eqiad.wmnet

The first 5 steps should be completed by the service owner that is returning the server to DC-ops (for reclaim to spare or decommissioning, dependent on server configuration and age.)

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system)
  • - unassign service owner from this task, check off completed steps, and assign to @RobH for followup on below steps.

Steps for DC-Ops:

The following steps cannot be interrupted, as it will leave the system in an unfinished state.

Start non-interrupt steps:

End non-interrupt steps.

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and change status of hardware to 'offline' when unracked.
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

restbase1009.eqiad.wmnet

The first 5 steps should be completed by the service owner that is returning the server to DC-ops (for reclaim to spare or decommissioning, dependent on server configuration and age.)

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system)
  • - unassign service owner from this task, check off completed steps, and assign to @RobH for followup on below steps.

Steps for DC-Ops:

The following steps cannot be interrupted, as it will leave the system in an unfinished state.

Start non-interrupt steps:

End non-interrupt steps.

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and change status of hardware to 'offline' when unracked.
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

restbase1010.eqiad.wmnet

The first 5 steps should be completed by the service owner that is returning the server to DC-ops (for reclaim to spare or decommissioning, dependent on server configuration and age.)

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system)
  • - unassign service owner from this task, check off completed steps, and assign to @RobH for followup on below steps.

Steps for DC-Ops:

The following steps cannot be interrupted, as it will leave the system in an unfinished state.

Start non-interrupt steps:

End non-interrupt steps.

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and change status of hardware to 'offline' when unracked.
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

restbase1011.eqiad.wmnet

The first 5 steps should be completed by the service owner that is returning the server to DC-ops (for reclaim to spare or decommissioning, dependent on server configuration and age.)

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system)
  • - unassign service owner from this task, check off completed steps, and assign to @RobH for followup on below steps.

Steps for DC-Ops:

The following steps cannot be interrupted, as it will leave the system in an unfinished state.

Start non-interrupt steps:

End non-interrupt steps.

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and change status of hardware to 'offline' when unracked.
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

restbase1012.eqiad.wmnet

The first 5 steps should be completed by the service owner that is returning the server to DC-ops (for reclaim to spare or decommissioning, dependent on server configuration and age.)

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system)
  • - unassign service owner from this task, check off completed steps, and assign to @RobH for followup on below steps.

Steps for DC-Ops:

The following steps cannot be interrupted, as it will leave the system in an unfinished state.

Start non-interrupt steps:

End non-interrupt steps.

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and change status of hardware to 'offline' when unracked.
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

restbase1013.eqiad.wmnet

The first 5 steps should be completed by the service owner that is returning the server to DC-ops (for reclaim to spare or decommissioning, dependent on server configuration and age.)

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system)
  • - unassign service owner from this task, check off completed steps, and assign to @RobH for followup on below steps.

Steps for DC-Ops:

The following steps cannot be interrupted, as it will leave the system in an unfinished state.

Start non-interrupt steps:

End non-interrupt steps.

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and change status of hardware to 'offline' when unracked.
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

restbase1014.eqiad.wmnet

The first 5 steps should be completed by the service owner that is returning the server to DC-ops (for reclaim to spare or decommissioning, dependent on server configuration and age.)

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system)
  • - unassign service owner from this task, check off completed steps, and assign to @RobH for followup on below steps.

Steps for DC-Ops:

The following steps cannot be interrupted, as it will leave the system in an unfinished state.

Start non-interrupt steps:

End non-interrupt steps.

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and change status of hardware to 'offline' when unracked.
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

restbase1015.eqiad.wmnet

The first 5 steps should be completed by the service owner that is returning the server to DC-ops (for reclaim to spare or decommissioning, dependent on server configuration and age.)

Steps for service owner:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp, replace with role(spare::system)
  • - unassign service owner from this task, check off completed steps, and assign to @RobH for followup on below steps.

Steps for DC-Ops:

The following steps cannot be interrupted, as it will leave the system in an unfinished state.

Start non-interrupt steps:

End non-interrupt steps.

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox with result and change status of hardware to 'offline' when unracked.
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
removing mgmt/asset tag of a decom serveroperations/dnsmaster+1 -1
DNS: Remove mgmt DNS for restbase100[7-9] and restbase101[0-5]operations/dnsmaster+1 -30
decom restbase10(0[7-9]|1[0-5])operations/puppetproduction+0 -133
decom restbase10(0[7-9]|1[0-5]) prod dnsoperations/dnsmaster+10 -72
Customize query in gerrit

Related Objects

Event Timeline

fgiunchedi created this task.Jun 27 2019, 1:30 PM
RobH moved this task from Backlog to Ready for Decommission on the decommission-hardware board.Jul 15 2019, 6:07 PM
Dzahn subscribed.EditedJul 22 2019, 10:39 PM

restbase1007.mgmt is alerting in Icinga but this ticket says restbase1007 was done.

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=restbase1007.mgmt&service=SSH+restbase1007.mgmt

ACKed

RobH added a comment.Jul 25 2019, 7:02 PM

Network port info:

restbase1007 asw2-a-eqiad:ge-4/0/30
restbase1010 asw2-a-eqiad:ge-3/0/16
restbase1011 asw2-a-eqiad:ge-3/0/17

restbase1012 asw2-c-eqiad:ge-4/0/28
restbase1013 asw2-c-eqiad:ge-4/0/29
restbase1008 asw2-c-eqiad:ge-5/0/31

restbase1014 asw2-d-eqiad:ge-4/0/20
restbase1015 asw2-d-eqiad:ge-4/0/21
restbase1009 asw2-d-eqiad:ge-3/0/26

ops-monitoring-bot added a comment.Jul 25 2019, 7:03 PM

cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: restbase[1007-1009].eqiad.wmnet

  • restbase1008.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
  • restbase1009.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
  • restbase1007.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
ops-monitoring-bot added a comment.Jul 25 2019, 7:04 PM

cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: restbase[1010-1015].eqiad.wmnet

  • restbase1010.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
  • restbase1015.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
  • restbase1011.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
  • restbase1012.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
  • restbase1013.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
  • restbase1014.eqiad.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
RobH updated the task description. (Show Details)Jul 25 2019, 7:05 PM
gerritbot added a comment.Jul 25 2019, 7:07 PM

Change 525627 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom restbase10(0[7-9]|1[0-5]) prod dns

https://gerrit.wikimedia.org/r/525627

gerritbot added a project: Patch-For-Review.Jul 25 2019, 7:07 PM
gerritbot added a comment.Jul 25 2019, 7:10 PM

Change 525628 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom restbase10(0[7-9]|1[0-5])

https://gerrit.wikimedia.org/r/525628

gerritbot added a comment.Jul 25 2019, 7:10 PM

Change 525627 merged by RobH:
[operations/dns@master] decom restbase10(0[7-9]|1[0-5]) prod dns

https://gerrit.wikimedia.org/r/525627

gerritbot added a comment.Jul 25 2019, 7:11 PM

Change 525628 merged by RobH:
[operations/puppet@production] decom restbase10(0[7-9]|1[0-5])

https://gerrit.wikimedia.org/r/525628

RobH updated the task description. (Show Details)Jul 25 2019, 7:14 PM
RobH edited projects, added ops-eqiad; removed Patch-For-Review.
Restricted Application added a project: SRE. · View Herald TranscriptJul 25 2019, 7:14 PM
RobH removed RobH as the assignee of this task.Jul 25 2019, 7:15 PM
RobH moved this task from Backlog to Decommission on the ops-eqiad board.
RobH moved this task from Ready for Decommission to pending onsite steps (eqiad) on the decommission-hardware board.
RobH mentioned this in T224268: Decommission rhenium.Jul 25 2019, 7:41 PM
MoritzMuehlenhoff mentioned this in T224549: Track remaining jessie systems in production.Aug 5 2019, 7:05 AM
• Cmjohnson added subscribers: Jclark-ctr, • Cmjohnson.Sep 19 2019, 8:21 PM

@Jclark-ctr wipe, remove the servers, update netbox and the google sheet. Please assign back to me once everything is complete

• Cmjohnson assigned this task to Jclark-ctr.Sep 19 2019, 8:21 PM
Papaul subscribed.Oct 8 2019, 12:54 AM
papaul@asw2-a-eqiad# show | compare 
[edit interfaces]
-   ge-3/0/16 {
-       description "restbase1010 1G";
-   }
-   ge-3/0/17 {
-       description "restbase1011 1G";
-   }
Papaul updated the task description. (Show Details)Oct 8 2019, 12:55 AM
Papaul updated the task description. (Show Details)Oct 8 2019, 12:58 AM
Papaul updated the task description. (Show Details)Oct 8 2019, 3:21 AM
papaul@asw2-a-eqiad# show | compare 
[edit interfaces]
-   ge-4/0/30 {
-       description restbase1007;
-   }
Jclark-ctr updated the task description. (Show Details)Oct 11 2019, 10:41 PM
Papaul added a comment.Oct 15 2019, 8:36 PM
papaul@asw2-c-eqiad# show | compare 
[edit interfaces]
-   ge-4/0/28 {
-       description restbase1012;
-   }
-   ge-4/0/29 {
-       description restbase1013;
-   }
-   ge-5/0/31 {
-       description restbase1008;
-   }
papaul@asw2-d-eqiad# show | compare        
[edit interfaces ge-3/0/26]
-   description restbase1009;
[edit interfaces ge-4/0/20]
-   description restbase1014;
[edit interfaces ge-4/0/21]
-   description restbase1015;
Papaul updated the task description. (Show Details)Oct 15 2019, 8:39 PM
gerritbot added a comment.Oct 15 2019, 9:36 PM

Change 543222 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for restbase100[7-9] and restbase101[0-5]

https://gerrit.wikimedia.org/r/543222

gerritbot added a project: Patch-For-Review.Oct 15 2019, 9:36 PM
gerritbot added a comment.Oct 15 2019, 10:29 PM

Change 543222 merged by Papaul:
[operations/dns@master] DNS: Remove mgmt DNS for restbase100[7-9] and restbase101[0-5]

https://gerrit.wikimedia.org/r/543222

Papaul closed this task as Resolved.Oct 15 2019, 10:31 PM
Papaul updated the task description. (Show Details)

complete

gerritbot added a comment.May 13 2020, 7:54 PM

Change 596276 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] removing mgmt/asset tag of a decom server

https://gerrit.wikimedia.org/r/596276

gerritbot added a comment.May 13 2020, 7:57 PM

Change 596276 merged by Cmjohnson:
[operations/dns@master] removing mgmt/asset tag of a decom server

https://gerrit.wikimedia.org/r/596276

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL