Page MenuHomePhabricator
Create Task
Maniphest T227565

decommission db2038
Closed, ResolvedPublicRequest
Actions

Description

This task will track the decommission-hardware of server db2038.codfw.wmnet

The first 5 steps should be completed by the service owner that is returning the server to DC-ops (for reclaim to spare or decommissioning, dependent on server configuration and age.)

db2038.codfw.wmnet

Steps for service owner:

Steps for DC-Ops:

The following steps cannot be interrupted, as it will leave the system in an unfinished state.

Start non-interrupt steps:

  • - disable puppet on host
  • - power down host
  • - update netbox status to Inventory (if decom) or Planned (if spare)
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate (handled by wmf-decommission-host)
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key (handled by wmf-decommission-host)

End non-interrupt steps.

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: system added back to spares tracking (by onsite)

Details

SubjectRepoBranchLines +/-
DNS: Remove mgmt DNS for db2038operations/dnsmaster+2 -4
wmnet: Remove production entries db2037-db2041operations/dnsmaster+0 -8
mariadb: Remove entries for db2037-db2041operations/puppetproduction+0 -20
mariadb: Decommission db2038operations/puppetproduction+6 -10
db-eqiad,db-codfw.php: Remove db2038operations/mediawiki-configmaster+0 -3
db2038: Disable notificationsoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui renamed this task from decommission db2035 to decommission db2038.Jul 9 2019, 8:47 AM
Marostegui created this task.
Marostegui mentioned this in T221533: Decommission old coredb machines (<=db2042).
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Jul 9 2019, 8:51 AM

Change 521435 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2038: Disable notifications

https://gerrit.wikimedia.org/r/521435

gerritbot added a project: Patch-For-Review.Jul 9 2019, 8:51 AM
gerritbot added a comment.Jul 9 2019, 8:54 AM

Change 521435 merged by Marostegui:
[operations/puppet@production] db2038: Disable notifications

https://gerrit.wikimedia.org/r/521435

Marostegui updated the task description. (Show Details)Jul 9 2019, 8:59 AM
Maintenance_bot removed a project: Patch-For-Review.Jul 9 2019, 9:10 AM
RobH moved this task from Backlog to Blocked on Service Owners on the decommission-hardware board.Jul 15 2019, 6:07 PM
gerritbot added a comment.Jul 29 2019, 7:26 AM

Change 526092 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2038

https://gerrit.wikimedia.org/r/526092

gerritbot added a project: Patch-For-Review.Jul 29 2019, 7:26 AM
Marostegui updated the task description. (Show Details)Jul 29 2019, 7:28 AM
gerritbot added a comment.Jul 29 2019, 7:28 AM

Change 526092 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2038

https://gerrit.wikimedia.org/r/526092

Maintenance_bot removed a project: Patch-For-Review.Jul 29 2019, 8:10 AM
Stashbot added a comment.Jul 30 2019, 8:10 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-30T08:10:07Z] <marostegui> Remove db2038 from tendril and zarcillo T227565

gerritbot added a comment.Jul 30 2019, 8:10 AM

Change 526382 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Decommission db2038

https://gerrit.wikimedia.org/r/526382

gerritbot added a project: Patch-For-Review.Jul 30 2019, 8:10 AM
gerritbot added a comment.Jul 30 2019, 8:14 AM

Change 526382 merged by Marostegui:
[operations/puppet@production] mariadb: Decommission db2038

https://gerrit.wikimedia.org/r/526382

Stashbot added a comment.Jul 30 2019, 8:17 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-30T08:17:09Z] <marostegui> Stop MySQL on db2038 T227565

Marostegui reassigned this task from Marostegui to RobH.Jul 30 2019, 8:20 AM
Marostegui updated the task description. (Show Details)
Marostegui edited projects, added ops-codfw; removed Patch-For-Review.
Restricted Application added a project: SRE. · View Herald TranscriptJul 30 2019, 8:20 AM
Marostegui added a comment.Jul 30 2019, 8:21 AM

This host is ready for DC-Ops to finish its decommission

Marostegui added a parent task: T221533: Decommission old coredb machines (<=db2042).Jul 30 2019, 8:24 AM
Papaul moved this task from Backlog to Decommission on the ops-codfw board.Aug 16 2019, 3:43 PM
Marostegui moved this task from Blocked on Service Owners to Ready for Decommission on the decommission-hardware board.Sep 5 2019, 9:40 AM
gerritbot added a comment.Sep 25 2019, 5:21 AM

Change 539016 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove entries for db2037-db2041

https://gerrit.wikimedia.org/r/539016

gerritbot added a project: Patch-For-Review.Sep 25 2019, 5:21 AM

Change 539017 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Remove production entries db2037-db2041

https://gerrit.wikimedia.org/r/539017

ops-monitoring-bot added a comment.Sep 25 2019, 5:23 AM

cookbooks.sre.hosts.decommission executed by marostegui@cumin1001 for hosts: db2038.codfw.wmnet

  • db2038.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Wiped bootloaders
    • Powered off
    • Set Netbox status to Decommissioning
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Sep 25 2019, 5:24 AM

Change 539016 merged by Marostegui:
[operations/puppet@production] mariadb: Remove entries for db2037-db2041

https://gerrit.wikimedia.org/r/539016

gerritbot added a comment.Sep 25 2019, 5:25 AM

Change 539017 merged by Marostegui:
[operations/dns@master] wmnet: Remove production entries db2037-db2041

https://gerrit.wikimedia.org/r/539017

Marostegui reassigned this task from RobH to Papaul.Sep 25 2019, 5:27 AM
Marostegui updated the task description. (Show Details)
Marostegui removed a project: Patch-For-Review.
Marostegui moved this task from Ready for Decommission to pending onsite steps (codfw) on the decommission-hardware board.

Host ready for @Papaul to decommission

Papaul added a comment.Sep 25 2019, 3:32 PM
papaul@asw-c-codfw# show | compare 
[edit interfaces interface-range vlan-private1-c-codfw]
-    member ge-6/0/5;
[edit interfaces interface-range disabled]
     member ge-6/0/4 { ... }
+    member ge-6/0/5;
[edit interfaces]
-   ge-6/0/5 {
-       description db2038;
-       enable;
-   }
Papaul updated the task description. (Show Details)Sep 25 2019, 3:32 PM
Papaul updated the task description. (Show Details)Sep 26 2019, 4:58 PM
Papaul updated the task description. (Show Details)Sep 30 2019, 8:04 PM
Papaul updated the task description. (Show Details)
gerritbot added a comment.Oct 1 2019, 12:45 AM

Change 539995 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for db2038

https://gerrit.wikimedia.org/r/539995

gerritbot added a project: Patch-For-Review.Oct 1 2019, 12:45 AM
gerritbot added a comment.Oct 1 2019, 1:07 AM

Change 539995 merged by Papaul:
[operations/dns@master] DNS: Remove mgmt DNS for db2038

https://gerrit.wikimedia.org/r/539995

Papaul closed this task as Resolved.Oct 1 2019, 1:11 AM
Papaul updated the task description. (Show Details)

This is complete

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL