Page MenuHomePhabricator
Create Task
Maniphest T223950

Decommission db2041
Closed, ResolvedPublic
Actions

Description

db2041 is ready to be decommissioned

db2041

Decommission Checklist

START NON-INTERRUPPTABLE STEPS - please assign to @RobH for the non-interrupt steps

  • - disable puppet on host
  • - power down host
  • - update status in netbox (inventory for decom, planned for spare)
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate (handled by wmf-decommission-host)
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key (handled by wmf-decommission-host)

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite) use hdparm for ssds and wipe for hdds
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
DNS: Remove mgmt DNS for db2041 db204[3-4] db204[6-7] and db2049operations/dnsmaster+1 -24
wmnet: Remove production entries db2037-db2041operations/dnsmaster+0 -8
mariadb: Remove entries for db2037-db2041operations/puppetproduction+0 -20
mariadb: Decommission db2041operations/puppetproduction+5 -10
db-eqiad,db-codfw.php: Remove db2041operations/mediawiki-configmaster+0 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Marostegui created this task.May 21 2019, 5:36 AM
gerritbot added a comment.May 21 2019, 5:37 AM

Change 511628 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2041

https://gerrit.wikimedia.org/r/511628

gerritbot added a project: Patch-For-Review.May 21 2019, 5:37 AM
gerritbot added a comment.May 21 2019, 5:40 AM

Change 511628 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2041

https://gerrit.wikimedia.org/r/511628

Stashbot added a comment.May 21 2019, 5:42 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-21T05:42:16Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2041 for decommissioning T223950 (duration: 00m 51s)

Stashbot added a comment.May 21 2019, 5:43 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-21T05:43:29Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2041 for decommissioning T223950 (duration: 00m 51s)

• Marostegui triaged this task as Medium priority.May 21 2019, 5:46 AM
• Marostegui updated the task description. (Show Details)
• Marostegui updated the task description. (Show Details)May 21 2019, 5:49 AM
Stashbot added a comment.May 21 2019, 5:50 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-21T05:50:23Z] <marostegui> Remove db2041 from tendril and zarcillo - T223950

gerritbot added a comment.May 21 2019, 5:56 AM

Change 511629 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Decommission db2041

https://gerrit.wikimedia.org/r/511629

Stashbot added a comment.May 21 2019, 8:56 AM

Mentioned in SAL (#wikimedia-operations) [2019-05-21T08:56:13Z] <marostegui> Stop MySQL on db2041 as it will be decommissioned T223950

• Marostegui moved this task from Triage to In progress on the DBA board.May 21 2019, 1:19 PM
gerritbot added a comment.May 21 2019, 2:48 PM

Change 511629 merged by Marostegui:
[operations/puppet@production] mariadb: Decommission db2041

https://gerrit.wikimedia.org/r/511629

• Marostegui assigned this task to RobH.May 22 2019, 5:17 AM
• Marostegui updated the task description. (Show Details)
• Marostegui edited projects, added ops-codfw, decommission-hardware; removed Patch-For-Review, DBA.
• Marostegui added a subscriber: Papaul.

This host is ready for DCOPs to take over.

Restricted Application added a project: SRE. · View Herald TranscriptMay 22 2019, 5:18 AM
• Marostegui mentioned this in T221533: Decommission old coredb machines (<=db2042).May 22 2019, 5:21 AM
RobH moved this task from Backlog to Ready for Decommission on the decommission-hardware board.Jun 12 2019, 8:30 AM
RobH moved this task from Backlog to Decommission on the ops-codfw board.Jun 12 2019, 1:11 PM
gerritbot added a comment.Sep 25 2019, 5:21 AM

Change 539016 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove entries for db2037-db2041

https://gerrit.wikimedia.org/r/539016

gerritbot added a project: Patch-For-Review.Sep 25 2019, 5:21 AM

Change 539017 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Remove production entries db2037-db2041

https://gerrit.wikimedia.org/r/539017

ops-monitoring-bot added a comment.Sep 25 2019, 5:24 AM

cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: db2041.codfw.wmnet

  • db2041.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Sep 25 2019, 5:24 AM

Change 539016 merged by Marostegui:
[operations/puppet@production] mariadb: Remove entries for db2037-db2041

https://gerrit.wikimedia.org/r/539016

gerritbot added a comment.Sep 25 2019, 5:26 AM

Change 539017 merged by Marostegui:
[operations/dns@master] wmnet: Remove production entries db2037-db2041

https://gerrit.wikimedia.org/r/539017

• Marostegui reassigned this task from RobH to Papaul.Sep 25 2019, 5:28 AM
• Marostegui removed a project: Patch-For-Review.
• Marostegui updated the task description. (Show Details)
• Marostegui moved this task from Ready for Decommission to pending onsite steps (codfw) on the decommission-hardware board.

Host ready for @Papaul to decommission

Papaul added a comment.Sep 25 2019, 3:44 PM
papaul@asw-c-codfw# show |compare 
[edit interfaces interface-range vlan-private1-c-codfw]
-    member ge-6/0/8;
[edit interfaces interface-range disabled]
     member ge-6/0/5 { ... }
+    member ge-6/0/8;
[edit interfaces]
-   ge-6/0/8 {
-       description db2041;
-       enable;
-   }
Papaul updated the task description. (Show Details)Sep 25 2019, 3:44 PM
Papaul updated the task description. (Show Details)Sep 26 2019, 5:24 PM
Papaul updated the task description. (Show Details)Sep 30 2019, 8:18 PM
gerritbot added a comment.Oct 1 2019, 10:18 PM

Change 540230 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for db2041 db204[3-4] db204[6-7] and db2049

https://gerrit.wikimedia.org/r/540230

gerritbot added a project: Patch-For-Review.Oct 1 2019, 10:18 PM
gerritbot added a comment.Oct 1 2019, 10:36 PM

Change 540230 merged by Papaul:
[operations/dns@master] DNS: Remove mgmt DNS for db2041 db204[3-4] db204[6-7] and db2049

https://gerrit.wikimedia.org/r/540230

Papaul updated the task description. (Show Details)Oct 1 2019, 10:38 PM

Complete

Papaul closed this task as Resolved.Oct 1 2019, 10:39 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits