Decommission db1037
Open, NormalPublic

Description

db1037's data was copied to db1098 so can now be decommissioned

  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/375777/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):
  • Added to spare role in site.pp until @Cmjohnson removes it forever: https://gerrit.wikimedia.org/r/#/c/375779/

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port & change switch port label to asset tag
  • - remove production dns entries & remove hostname entries in mgmt dns
  • - puppet node clean, puppet node deactivate, salt key removed

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - remove hostname label, remove hostname from visible label field in racktables (by onsite)
  • - system added back to spares tracking (by onsite)
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Sep 4, 8:54 AM
Marostegui triaged this task as Normal priority.Mon, Sep 4, 8:55 AM
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui updated the task description. (Show Details)Mon, Sep 4, 11:40 AM

Change 375777 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1037

https://gerrit.wikimedia.org/r/375777

Change 375779 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove db1037 for decomm

https://gerrit.wikimedia.org/r/375779

Change 375777 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1037

https://gerrit.wikimedia.org/r/375777

Mentioned in SAL (#wikimedia-operations) [2017-09-04T11:52:16Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)

Mentioned in SAL (#wikimedia-operations) [2017-09-04T11:53:17Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1037 as it will be decommissioned - T174902 (duration: 00m 46s)

Marostegui updated the task description. (Show Details)Mon, Sep 4, 12:08 PM

Change 375779 merged by Marostegui:
[operations/puppet@production] mariadb: Remove db1037 for decomm

https://gerrit.wikimedia.org/r/375779

Marostegui updated the task description. (Show Details)Mon, Sep 4, 12:17 PM
Marostegui added a subscriber: Cmjohnson.

Mentioned in SAL (#wikimedia-operations) [2017-09-04T12:18:15Z] <marostegui> Stop MySQL on db1037 as it is going to be decommissioned - T174902

Change 375790 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s6.hosts: Remove db1037 for decommission

https://gerrit.wikimedia.org/r/375790

Marostegui moved this task from In progress to Done on the DBA board.

MySQL is stopped.
This host is now ready for the remaining DC Ops steps to be completed

Change 375790 merged by jenkins-bot:
[operations/software@master] s6.hosts: Remove db1037 for decommission

https://gerrit.wikimedia.org/r/375790

Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Tue, Sep 5, 7:04 PM