Page MenuHomePhabricator

Decommission db2037
Closed, ResolvedPublic

Description

db2037 can be decommissioned.

db2037

Decommission Checklist

START NON-INTERRUPPTABLE STEPS - please assign to @RobH for the non-interrupt steps

  • - disable puppet on host
  • - power down host
  • - update status in netbox (inventory for decom, planned for spare)
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate (handled by wmf-decommission-host)
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key (handled by wmf-decommission-host)

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite) use hdparm for ssds and wipe for hdds
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

Related Gerrit Patches:
operations/dns : masterDNS: Remove mgmt DNS for db2037
operations/dns : masterwmnet: Remove production entries db2037-db2041
operations/puppet : productionmariadb: Remove entries for db2037-db2041
operations/puppet : productiondb2037: Prepare for decommission

Event Timeline

Restricted Application added a project: Operations. · View Herald TranscriptMay 31 2019, 9:52 AM

Change 513573 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2037: Prepare for decommission

https://gerrit.wikimedia.org/r/513573

Change 513573 merged by Marostegui:
[operations/puppet@production] db2037: Prepare for decommission

https://gerrit.wikimedia.org/r/513573

Mentioned in SAL (#wikimedia-operations) [2019-06-03T05:04:49Z] <marostegui> Stop MySQL on db2037 for decommission T224720

Mentioned in SAL (#wikimedia-operations) [2019-06-03T05:05:08Z] <marostegui> Remove db2037 from tendril and zarcillo T224720

Marostegui reassigned this task from Marostegui to RobH.Jun 3 2019, 5:06 AM
Marostegui triaged this task as Medium priority.
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)

This host is ready for DCOPs to take over.

RobH moved this task from Backlog to Decommission on the ops-codfw board.Jun 12 2019, 1:10 PM

Change 539016 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove entries for db2037-db2041

https://gerrit.wikimedia.org/r/539016

Change 539017 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Remove production entries db2037-db2041

https://gerrit.wikimedia.org/r/539017

cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: db2037.codfw.wmnet

  • db2037.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

Change 539016 merged by Marostegui:
[operations/puppet@production] mariadb: Remove entries for db2037-db2041

https://gerrit.wikimedia.org/r/539016

Change 539017 merged by Marostegui:
[operations/dns@master] wmnet: Remove production entries db2037-db2041

https://gerrit.wikimedia.org/r/539017

Marostegui reassigned this task from RobH to Papaul.Sep 25 2019, 5:26 AM
Marostegui removed a project: Patch-For-Review.
Marostegui updated the task description. (Show Details)

Host ready for @Papaul to decommission

papaul@asw-c-codfw# show | compare 
[edit interfaces interface-range vlan-private1-c-codfw]
-    member ge-6/0/4;
[edit interfaces interface-range disabled]
     member ge-6/0/3 { ... }
+    member ge-6/0/4;
[edit interfaces]
-   ge-6/0/4 {
-       description db2037;
-       enable;
-   }
Papaul updated the task description. (Show Details)Sep 25 2019, 3:28 PM
Papaul updated the task description. (Show Details)Sep 26 2019, 4:43 PM
Papaul updated the task description. (Show Details)Sep 30 2019, 8:03 PM

Change 539993 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for db2037

https://gerrit.wikimedia.org/r/539993

Change 539993 merged by Dzahn:
[operations/dns@master] DNS: Remove mgmt DNS for db2037

https://gerrit.wikimedia.org/r/539993

Papaul closed this task as Resolved.Oct 1 2019, 12:31 AM
Papaul updated the task description. (Show Details)

This is complete