Page MenuHomePhabricator

Decommission db1045
Closed, ResolvedPublic

Description

db1045 is ready to be decommissioned after its data was copied to db1099

  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/375748/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):
  • Added as spare to site.pp until @Cmjohnson removes it from good once he starts the DC Ops steps: https://gerrit.wikimedia.org/r/#/c/375750/

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port & change switch port label to asset tag
  • - remove production dns entries & remove hostname entries in mgmt dns
  • - puppet node clean, puppet node deactivate, salt key removed

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - remove hostname label, remove hostname from visible label field in racktables (by onsite)
  • - system added back to spares tracking (by onsite)

Event Timeline

Restricted Application added a project: Operations. · View Herald TranscriptSep 1 2017, 4:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui triaged this task as Medium priority.Sep 1 2017, 4:31 PM
Marostegui moved this task from Triage to Next on the DBA board.
Marostegui updated the task description. (Show Details)Sep 4 2017, 8:28 AM

Change 375748 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1045

https://gerrit.wikimedia.org/r/375748

Change 375750 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove db1045 - will be decommissioned

https://gerrit.wikimedia.org/r/375750

Change 375748 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1045

https://gerrit.wikimedia.org/r/375748

Mentioned in SAL (#wikimedia-operations) [2017-09-04T08:39:45Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 48s)

Mentioned in SAL (#wikimedia-operations) [2017-09-04T08:40:39Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1045 as it will be decommissioned - T174806 (duration: 00m 46s)

Change 375750 merged by Marostegui:
[operations/puppet@production] mariadb: Remove db1045 - will be decommissioned

https://gerrit.wikimedia.org/r/375750

Marostegui updated the task description. (Show Details)Sep 4 2017, 8:47 AM
Marostegui added a subscriber: Cmjohnson.

Mentioned in SAL (#wikimedia-operations) [2017-09-04T08:48:41Z] <marostegui> Stop MySQL on db1045 as it will be decommissioned - T174806

Marostegui moved this task from Next to Done on the DBA board.

This server is ready to be totally decommissioned and only pending the DC Ops steps, so assigning it to @Cmjohnson for the pending steps

Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Sep 5 2017, 7:27 PM
Cmjohnson updated the task description. (Show Details)Dec 12 2017, 5:45 PM
Cmjohnson updated the task description. (Show Details)Dec 19 2017, 8:19 PM

This host still shows up in puppetdb, i.e. misses the deactivate step (e.g. visible in https://servermon.wikimedia.org/hosts/)

Cmjohnson closed this task as Resolved.Jan 10 2018, 3:48 PM