Page MenuHomePhabricator
Create Task
Maniphest T224720

Decommission db2037
Closed, ResolvedPublic
Actions

Description

db2037 can be decommissioned.

db2037

Decommission Checklist

START NON-INTERRUPPTABLE STEPS - please assign to @RobH for the non-interrupt steps

  • - disable puppet on host
  • - power down host
  • - update status in netbox (inventory for decom, planned for spare)
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate (handled by wmf-decommission-host)
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key (handled by wmf-decommission-host)

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite) use hdparm for ssds and wipe for hdds
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
DNS: Remove mgmt DNS for db2037operations/dnsmaster+0 -4
wmnet: Remove production entries db2037-db2041operations/dnsmaster+0 -8
mariadb: Remove entries for db2037-db2041operations/puppetproduction+0 -20
db2037: Prepare for decommissionoperations/puppetproduction+6 -9
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Marostegui created this task.May 31 2019, 9:52 AM
Restricted Application added a project: SRE. · View Herald TranscriptMay 31 2019, 9:52 AM
gerritbot added a comment.May 31 2019, 9:54 AM

Change 513573 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2037: Prepare for decommission

https://gerrit.wikimedia.org/r/513573

gerritbot added a project: Patch-For-Review.May 31 2019, 9:54 AM
gerritbot added a comment.Jun 3 2019, 5:03 AM

Change 513573 merged by Marostegui:
[operations/puppet@production] db2037: Prepare for decommission

https://gerrit.wikimedia.org/r/513573

Stashbot added a comment.Jun 3 2019, 5:04 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-03T05:04:49Z] <marostegui> Stop MySQL on db2037 for decommission T224720

Stashbot added a comment.Jun 3 2019, 5:05 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-03T05:05:08Z] <marostegui> Remove db2037 from tendril and zarcillo T224720

• Marostegui reassigned this task from • Marostegui to RobH.Jun 3 2019, 5:06 AM
• Marostegui triaged this task as Medium priority.
• Marostegui updated the task description. (Show Details)
• Marostegui updated the task description. (Show Details)

This host is ready for DCOPs to take over.

• Marostegui mentioned this in T221533: Decommission old coredb machines (<=db2042).Jun 3 2019, 5:07 AM
Maintenance_bot removed a project: Patch-For-Review.Jun 3 2019, 5:10 AM
RobH moved this task from Backlog to Ready for Decommission on the decommission-hardware board.Jun 12 2019, 8:30 AM
RobH moved this task from Backlog to Decommission on the ops-codfw board.Jun 12 2019, 1:10 PM
gerritbot added a comment.Sep 25 2019, 5:21 AM

Change 539016 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove entries for db2037-db2041

https://gerrit.wikimedia.org/r/539016

gerritbot added a project: Patch-For-Review.Sep 25 2019, 5:21 AM

Change 539017 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Remove production entries db2037-db2041

https://gerrit.wikimedia.org/r/539017

ops-monitoring-bot added a comment.Sep 25 2019, 5:23 AM

cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: db2037.codfw.wmnet

  • db2037.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Sep 25 2019, 5:24 AM

Change 539016 merged by Marostegui:
[operations/puppet@production] mariadb: Remove entries for db2037-db2041

https://gerrit.wikimedia.org/r/539016

gerritbot added a comment.Sep 25 2019, 5:25 AM

Change 539017 merged by Marostegui:
[operations/dns@master] wmnet: Remove production entries db2037-db2041

https://gerrit.wikimedia.org/r/539017

• Marostegui reassigned this task from RobH to Papaul.Sep 25 2019, 5:26 AM
• Marostegui removed a project: Patch-For-Review.
• Marostegui updated the task description. (Show Details)
• Marostegui moved this task from Ready for Decommission to pending onsite steps (codfw) on the decommission-hardware board.

Host ready for @Papaul to decommission

Papaul added a comment.Sep 25 2019, 3:27 PM
papaul@asw-c-codfw# show | compare 
[edit interfaces interface-range vlan-private1-c-codfw]
-    member ge-6/0/4;
[edit interfaces interface-range disabled]
     member ge-6/0/3 { ... }
+    member ge-6/0/4;
[edit interfaces]
-   ge-6/0/4 {
-       description db2037;
-       enable;
-   }
Papaul updated the task description. (Show Details)Sep 25 2019, 3:28 PM
Papaul updated the task description. (Show Details)Sep 26 2019, 4:43 PM
Papaul updated the task description. (Show Details)Sep 30 2019, 8:03 PM
gerritbot added a comment.Oct 1 2019, 12:02 AM

Change 539993 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for db2037

https://gerrit.wikimedia.org/r/539993

gerritbot added a project: Patch-For-Review.Oct 1 2019, 12:02 AM
gerritbot added a comment.Oct 1 2019, 12:28 AM

Change 539993 merged by Dzahn:
[operations/dns@master] DNS: Remove mgmt DNS for db2037

https://gerrit.wikimedia.org/r/539993

Papaul closed this task as Resolved.Oct 1 2019, 12:31 AM
Papaul updated the task description. (Show Details)

This is complete

Maintenance_bot removed a project: Patch-For-Review.Oct 1 2019, 1:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL