Page MenuHomePhabricator

Decommission db1021
Closed, ResolvedPublic

Description

db1021's data has been copied over db1053. So it is ready to be decommissioned

  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/393524/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port & change switch port label to asset tag
  • - remove production dns entries & remove hostname entries in mgmt dns
  • - puppet node clean, puppet node deactivate, salt key removed

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - remove hostname label, remove hostname from visible label field in racktables (by onsite)
  • - system added back to spares tracking (by onsite)

Event Timeline

Change 393524 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1021

https://gerrit.wikimedia.org/r/393524

Marostegui moved this task from Triage to In progress on the DBA board.Nov 27 2017, 6:48 AM

Change 393524 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1021

https://gerrit.wikimedia.org/r/393524

Mentioned in SAL (#wikimedia-operations) [2017-11-27T06:52:00Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1021 from the config as it will be decommissioned - T181378 (duration: 00m 45s)

Mentioned in SAL (#wikimedia-operations) [2017-11-27T06:53:03Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1021 from the config as it will be decommissioned - T181378 (duration: 00m 44s)

Change 393525 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Prepare to decommission db1021

https://gerrit.wikimedia.org/r/393525

Change 393525 merged by Marostegui:
[operations/puppet@production] mariadb: Prepare to decommission db1021

https://gerrit.wikimedia.org/r/393525

Change 393528 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s2.hosts: Remove db1021

https://gerrit.wikimedia.org/r/393528

Change 393528 merged by jenkins-bot:
[operations/software@master] s2.hosts: Remove db1021

https://gerrit.wikimedia.org/r/393528

Marostegui updated the task description. (Show Details)Nov 27 2017, 7:09 AM
Marostegui updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2017-11-27T07:10:41Z] <marostegui> Stop MySQL on db1021 as it will be decommissioned - T181378

Marostegui moved this task from In progress to Done on the DBA board.

This server is now ready for @Cmjohnson to take over and fully decommission it.
Thanks!

Cmjohnson updated the task description. (Show Details)Dec 12 2017, 5:44 PM

Still showing in servermon, also seems like a missing "puppet node deactivate"

Cmjohnson updated the task description. (Show Details)Dec 19 2017, 8:18 PM
Cmjohnson closed this task as Resolved.Jan 10 2018, 3:49 PM