Page MenuHomePhabricator
Create Task
Maniphest T228281

decommission db2045.codfw.wmnet
Closed, ResolvedPublicRequest
Actions

Description

This task will track the decommission-hardware of server db2045.codfw.wmnet

The first 5 steps should be completed by the service owner that is returning the server to DC-ops (for reclaim to spare or decommissioning, dependent on server configuration and age.)

db2045

Steps for service owner:

Steps for DC-Ops:

The following steps cannot be interrupted, as it will leave the system in an unfinished state.

Start non-interrupt steps:

  • - disable puppet on host
  • - power down host
  • - update netbox status to decommissioning
  • - disable switch port
  • - switch port assignment noted on this task (for later removal) asw-b-codfw:ge-3/0/20
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate (handled by wmf-decommission-host)
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key (handled by wmf-decommission-host)

End non-interrupt steps.

  • - Label BBU as broken so it doesn't get re-used T227862
  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update netbox including status to offline
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
DNS: Remove mgmt DNS for db2045operations/dnsmaster+0 -4
db2045 decomoperations/puppetproduction+0 -5
db2045 prod entry removaloperations/dnsmaster+1 -2
db2045 prod dns decomoperations/dnsmaster+1 -2
mariadb: Set db2045 to spare for decommissioningoperations/puppetproduction+6 -9
db-eqiad,db-codfw.php: Remove db2045, host will be decommissionedoperations/mediawiki-configmaster+0 -3
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT208323 Predictive failures on disk S.M.A.R.T. status
 ResolvedMarosteguiT228258 Decommission db2043-db2070
 ResolvedRequestPapaulT228281 decommission db2045.codfw.wmnet

Event Timeline

Marostegui created this task.Jul 17 2019, 1:56 PM
Marostegui mentioned this in T227862: (OoW) db2045 failed battery.
Marostegui added a parent task: T228258: Decommission db2043-db2070.
Marostegui mentioned this in T228258: Decommission db2043-db2070.
Marostegui moved this task from Triage to In progress on the DBA board.Jul 17 2019, 2:01 PM
gerritbot added a comment.Jul 18 2019, 5:05 AM

Change 524081 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2045, host will be decommissioned

https://gerrit.wikimedia.org/r/524081

gerritbot added a project: Patch-For-Review.Jul 18 2019, 5:05 AM

Change 524081 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2045, host will be decommissioned

https://gerrit.wikimedia.org/r/524081

Stashbot added a comment.Jul 18 2019, 5:08 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-18T05:08:38Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db2045 from config, will be decommissioned T228281 (duration: 00m 56s)

Stashbot added a comment.Jul 18 2019, 5:09 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-18T05:09:39Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db2045 from config, will be decommissioned T228281 (duration: 00m 54s)

Maintenance_bot removed a project: Patch-For-Review.Jul 18 2019, 5:10 AM
gerritbot added a comment.Jul 18 2019, 5:11 AM

Change 524082 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Set db2045 to spare for decommissioning

https://gerrit.wikimedia.org/r/524082

gerritbot added a project: Patch-For-Review.Jul 18 2019, 5:11 AM

Change 524082 merged by Marostegui:
[operations/puppet@production] mariadb: Set db2045 to spare for decommissioning

https://gerrit.wikimedia.org/r/524082

Stashbot added a comment.Jul 18 2019, 5:16 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-18T05:16:12Z] <marostegui> Disable notifications on db2045 T228281

Stashbot added a comment.Jul 18 2019, 5:18 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-18T05:18:49Z] <marostegui> Remove db2045 from tendril and zarcillo T228281

Stashbot added a comment.Jul 18 2019, 5:22 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-18T05:22:23Z] <marostegui> Stop MySQL on db2045, host will be decommissioned T228281

Marostegui reassigned this task from Marostegui to RobH.Jul 18 2019, 5:24 AM
Marostegui updated the task description. (Show Details)
Marostegui removed projects: Patch-For-Review, DBA.

This host is ready for DC-Ops to start their decommissioning steps

Marostegui added a project: ops-codfw.Jul 18 2019, 3:04 PM
Restricted Application added a project: SRE. · View Herald TranscriptJul 18 2019, 3:04 PM
RobH moved this task from Backlog to Ready for Decommission on the decommission-hardware board.Jul 25 2019, 3:09 PM
Papaul moved this task from Backlog to Decommission on the ops-codfw board.Aug 16 2019, 3:43 PM
RobH updated the task description. (Show Details)Aug 23 2019, 11:02 PM
ops-monitoring-bot added a comment.Aug 23 2019, 11:03 PM

cookbooks.sre.hosts.decommission executed by robh@cumin1001 for hosts: db2045.codfw.wmnet

  • db2045.codfw.wmnet
    • Removed from Puppet master and PuppetDB
    • Downtimed host on Icinga
    • Downtimed management interface on Icinga
    • Removed from DebMonitor
gerritbot added a comment.Aug 23 2019, 11:06 PM

Change 531979 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] db2045 prod dns decom

https://gerrit.wikimedia.org/r/531979

gerritbot added a project: Patch-For-Review.Aug 23 2019, 11:06 PM

Change 531979 abandoned by RobH:
db2045 prod dns decom

https://gerrit.wikimedia.org/r/531979

gerritbot added a comment.Aug 23 2019, 11:08 PM

Change 531980 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] db2045 prod entry removal

https://gerrit.wikimedia.org/r/531980

gerritbot added a comment.Aug 23 2019, 11:09 PM

Change 531980 merged by RobH:
[operations/dns@master] db2045 prod entry removal

https://gerrit.wikimedia.org/r/531980

gerritbot added a comment.Aug 23 2019, 11:09 PM

Change 531981 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] db2045 decom

https://gerrit.wikimedia.org/r/531981

gerritbot added a comment.Aug 23 2019, 11:10 PM

Change 531981 merged by RobH:
[operations/puppet@production] db2045 decom

https://gerrit.wikimedia.org/r/531981

Maintenance_bot removed a project: Patch-For-Review.Aug 23 2019, 11:10 PM
RobH reassigned this task from RobH to Papaul.Aug 23 2019, 11:11 PM
RobH updated the task description. (Show Details)
RobH moved this task from Ready for Decommission to pending onsite steps (codfw) on the decommission-hardware board.
Papaul added a comment.Aug 29 2019, 3:03 PM
papaul@asw-b-codfw# run show interfaces ge-3/0/20 descriptions 
Interface       Admin Link Description
ge-3/0/20       down  down DISABLED
Papaul updated the task description. (Show Details)Aug 29 2019, 3:03 PM
gerritbot added a comment.Aug 29 2019, 4:07 PM

Change 533255 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for db2045

https://gerrit.wikimedia.org/r/533255

gerritbot added a project: Patch-For-Review.Aug 29 2019, 4:07 PM

Change 533255 merged by Papaul:
[operations/dns@master] DNS: Remove mgmt DNS for db2045

https://gerrit.wikimedia.org/r/533255

Papaul closed this task as Resolved.Aug 29 2019, 4:09 PM
Papaul updated the task description. (Show Details)

Complete

Maintenance_bot removed a project: Patch-For-Review.Aug 29 2019, 4:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits