Page MenuHomePhabricator

Decommission db1044
Closed, ResolvedPublic

Description

db1044 is ready to be decommissioned:

  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/393747/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):
  • Host set to spare: https://gerrit.wikimedia.org/r/#/c/394919/

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port & change switch port label to asset tag
  • - remove production dns entries & remove hostname entries in mgmt dns
  • - puppet node clean, puppet node deactivate, salt key removed

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - remove hostname label, remove hostname from visible label field in racktables (by onsite)
  • - system added back to spares tracking (by onsite)

Event Timeline

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptNov 30 2017, 7:45 AM
Marostegui moved this task from Triage to In progress on the DBA board.Nov 30 2017, 7:48 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-04T06:38:48Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1044 from config as it will be decommissioned - T181696 (duration: 00m 45s)

Marostegui updated the task description. (Show Details)Dec 4 2017, 6:39 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-04T06:39:46Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1044 from config as it will be decommissioned - T181696 (duration: 00m 45s)

Change 394919 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Decommission db1044

https://gerrit.wikimedia.org/r/394919

Change 394920 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s3.hosts: Remove db1044

https://gerrit.wikimedia.org/r/394920

Change 394920 merged by jenkins-bot:
[operations/software@master] s3.hosts: Remove db1044

https://gerrit.wikimedia.org/r/394920

Change 394919 merged by Marostegui:
[operations/puppet@production] mariadb: Decommission db1044

https://gerrit.wikimedia.org/r/394919

Marostegui updated the task description. (Show Details)Dec 4 2017, 7:08 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-04T07:08:38Z] <marostegui> Stop MySQL on db1044 as it will be decommissioned - T181696

Marostegui reassigned this task from Marostegui to Cmjohnson.Dec 4 2017, 7:10 AM
Marostegui moved this task from In progress to Done on the DBA board.

This host is now fully ready to be decommissioned.

Cmjohnson lowered the priority of this task from Medium to Low.Dec 12 2017, 3:00 PM
Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.
Cmjohnson updated the task description. (Show Details)

Disks are wiped

Cmjohnson updated the task description. (Show Details)Dec 19 2017, 8:18 PM

This host still shows up in puppetdb, i.e. misses the deactivate step (e.g. visible in https://servermon.wikimedia.org/hosts/)

Cmjohnson closed this task as Resolved.Jan 10 2018, 3:47 PM