Page MenuHomePhabricator
Create Task
Maniphest T209181

Decommission rdb1001, rdb1002, rdb1003, rdb1004, rdb1007, rdb1008
Closed, ResolvedPublic
Actions

Description

rdb1001:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration (N/A)
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - system unracked and decommissioned (by onsite), update netbox with result
  • - switch port configration removed from switch once system is unracked.
  • - add system to decommission tracking google sheet
  • - mgmt dns entries removed.

rdb1002:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration (N/A)
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - system unracked and decommissioned (by onsite), update netbox with result
  • - switch port configration removed from switch once system is unracked.
  • - add system to decommission tracking google sheet
  • - mgmt dns entries removed.

rdb1003:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration (N/A)
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - system unracked and decommissioned (by onsite), update netbox with result
  • - switch port configration removed from switch once system is unracked.
  • - add system to decommission tracking google sheet
  • - mgmt dns entries removed.

rdb1004:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration (N/A)
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - system unracked and decommissioned (by onsite), update netbox with result
  • - switch port configration removed from switch once system is unracked.
  • - add system to decommission tracking google sheet
  • - mgmt dns entries removed.

rdb1007:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration (N/A)
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - system unracked and decommissioned (by onsite), update netbox with result
  • - switch port configration removed from switch once system is unracked.
  • - add system to decommission tracking google sheet
  • - mgmt dns entries removed.

rdb1008:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration (N/A)
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - system unracked and decommissioned (by onsite), update netbox with result
  • - switch port configration removed from switch once system is unracked.
  • - add system to decommission tracking google sheet
  • - mgmt dns entries removed.
In T209181#5011725, > NETWORK PORT INFO

rdb1001:asw2-c-eqiad:ge-4/0/9
rdb1002:asw2-c-eqiad:ge-7/0/18
rdb1003:asw-a-eqiad:ge-4/0/43
rdb1004:asw2-b-eqiad:ge-4/0/43
rdb1007:asw2-c-eqiad:ge-4/0/3
rdb1008: asw2-c-eqiad:ge-5/0/30

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -48decom of rdb100[123478]
operations/dnsmaster+2 -13decom rdb100[123478].eqiad.wmnet dns entries
operations/puppetproduction+0 -4Remove obsolete Hiera files
operations/puppetproduction+7 -7Reimage rdb2003/rdb2004, switch rdb100[123478] to spare::system
Customize query in gerrit

Related Objects
Search...

Event Timeline

jijiki triaged this task as Medium priority.Nov 9 2018, 8:44 PM
jijiki created this task.
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptNov 9 2018, 8:44 PM
gerritbot added a comment.Nov 9 2018, 8:44 PM

Change 472714 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Reimage rdb2003/rdb2004, switch rdb100[123478] to spare::system

https://gerrit.wikimedia.org/r/472714

gerritbot added a project: Patch-For-Review.Nov 9 2018, 8:44 PM
jijiki updated the task description. (Show Details)Nov 9 2018, 8:45 PM
jijiki added a project: decommission-hardware.Nov 9 2018, 8:52 PM
gerritbot added a comment.Nov 9 2018, 8:55 PM

Change 472714 merged by Effie Mouzeli:
[operations/puppet@production] Reimage rdb2003/rdb2004, switch rdb100[123478] to spare::system

https://gerrit.wikimedia.org/r/472714

jijiki moved this task from Inbox 🐅 to In Progress 🏋️‍♀️ on the User-jijiki board.Nov 12 2018, 11:29 AM
RobH updated the task description. (Show Details)Nov 13 2018, 9:55 PM
jijiki updated the task description. (Show Details)Nov 14 2018, 6:40 PM
jijiki removed a project: User-Joe.Nov 19 2018, 9:06 AM
jijiki moved this task from In Progress 🏋️‍♀️ to Misc on the User-jijiki board.Nov 19 2018, 9:10 AM
jijiki removed a subscriber: jijiki.Jan 3 2019, 8:44 AM
gerritbot added a comment.Jan 24 2019, 3:43 PM

Change 482295 had a related patch set uploaded (by Effie Mouzeli; owner: Muehlenhoff):
[operations/puppet@production] Remove obsolete Hiera files

https://gerrit.wikimedia.org/r/482295

gerritbot added a comment.Jan 24 2019, 3:47 PM

Change 482295 merged by Effie Mouzeli:
[operations/puppet@production] Remove obsolete Hiera files

https://gerrit.wikimedia.org/r/482295

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:38 PM
RobH moved this task from Backlog to Ready for Decommission on the decommission-hardware board.Mar 7 2019, 8:06 PM
jijiki updated the task description. (Show Details)Mar 8 2019, 5:26 PM
Stashbot added a comment.Mar 8 2019, 5:30 PM

Mentioned in SAL (#wikimedia-operations) [2019-03-08T17:30:00Z] <robh> decom in progress for rdb100[123478] via T209181

RobH edited projects, added ops-eqiad; removed Patch-For-Review.Mar 8 2019, 5:47 PM
RobH updated the task description. (Show Details)
RobH moved this task from Backlog to Decommission on the ops-eqiad board.
RobH added a subscriber: RobH.Mar 8 2019, 6:03 PM

NETWORK PORT INFO

rdb1001:asw2-c-eqiad:ge-4/0/9
rdb1002:asw2-c-eqiad:ge-7/0/18
rdb1003:asw-a-eqiad:ge-4/0/43
rdb1004:asw2-b-eqiad:ge-4/0/43
rdb1007:asw2-c-eqiad:ge-4/0/3
rdb1008: asw2-c-eqiad:ge-5/0/30

ops-monitoring-bot added a comment.Mar 8 2019, 6:03 PM

wmf-decommission-host was executed by robh for rdb1001.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
ops-monitoring-bot added a comment.Mar 8 2019, 6:03 PM

wmf-decommission-host was executed by robh for rdb1002.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
ops-monitoring-bot added a comment.Mar 8 2019, 6:04 PM

wmf-decommission-host was executed by robh for rdb1003.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
ops-monitoring-bot added a comment.Mar 8 2019, 6:04 PM

wmf-decommission-host was executed by robh for rdb1004.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
ops-monitoring-bot added a comment.Mar 8 2019, 6:04 PM

wmf-decommission-host was executed by robh for rdb1007.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
ops-monitoring-bot added a comment.Mar 8 2019, 6:04 PM

wmf-decommission-host was executed by robh for rdb1008.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
gerritbot added a comment.Mar 8 2019, 6:06 PM

Change 495274 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom rdb100[123478].eqiad.wmnet dns entries

https://gerrit.wikimedia.org/r/495274

gerritbot added a project: Patch-For-Review.Mar 8 2019, 6:06 PM
gerritbot added a comment.Mar 8 2019, 6:08 PM

Change 495274 merged by RobH:
[operations/dns@master] decom rdb100[123478].eqiad.wmnet dns entries

https://gerrit.wikimedia.org/r/495274

gerritbot added a comment.Mar 8 2019, 6:09 PM

Change 495275 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom of rdb100[123478]

https://gerrit.wikimedia.org/r/495275

RobH removed a project: Patch-For-Review.Mar 8 2019, 6:10 PM
RobH updated the task description. (Show Details)
gerritbot added a comment.Mar 8 2019, 6:11 PM

Change 495275 merged by RobH:
[operations/puppet@production] decom of rdb100[123478]

https://gerrit.wikimedia.org/r/495275

RobH assigned this task to Cmjohnson.Mar 8 2019, 6:12 PM
RobH moved this task from Ready for Decommission to pending onsite steps (eqiad) on the decommission-hardware board.
Jclark-ctr updated the task description. (Show Details)Aug 1 2019, 3:34 PM
Jclark-ctr updated the task description. (Show Details)Aug 1 2019, 4:46 PM
Jclark-ctr updated the task description. (Show Details)Aug 21 2019, 5:58 PM
Jclark-ctr claimed this task.Oct 11 2019, 10:31 PM
Jclark-ctr added a subscriber: Cmjohnson.
Jclark-ctr updated the task description. (Show Details)Oct 11 2019, 10:58 PM
Jclark-ctr updated the task description. (Show Details)Oct 11 2019, 11:00 PM
Jclark-ctr updated the task description. (Show Details)Nov 28 2019, 12:06 AM
Papaul added a subscriber: Papaul.Jan 3 2020, 11:03 PM
papaul@asw2-c-eqiad# show | compare 
[edit interfaces]
-   ge-5/0/30 {
-       description rdb1008;
-   }
papaul@asw2-c-eqiad# show | compare 
[edit interfaces]
-   ge-4/0/3 {
-       description rdb1007;
-   }
RobH removed a subscriber: RobH.Mar 3 2020, 6:18 PM
RobH edited subscribers, added: RobH; removed: Papaul.Apr 1 2020, 5:07 PM
RobH updated the task description. (Show Details)Apr 1 2020, 5:36 PM
RobH removed subscribers: RobH, gerritbot.
Cmjohnson closed this task as Resolved.May 13 2020, 6:38 PM

verified all servers are gone, they are on the list that was sold already.

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL