Page MenuHomePhabricator

Decommission db1034
Closed, ResolvedPublic

Description

db1034 is ready to be decommissioned after transferring its data to the new multi-instance hosts in s7

  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/398046/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):
  • Host set to spare: https://gerrit.wikimedia.org/r/#/c/398425/

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port & change switch port label to asset tag
  • - remove production dns entries & remove hostname entries in mgmt dns
  • - puppet node clean, puppet node deactivate, salt key removed

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Related Objects

Event Timeline

Marostegui triaged this task as Medium priority.Dec 11 2017, 7:23 AM
Marostegui created this task.
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptDec 11 2017, 7:23 AM
Marostegui moved this task from Triage to In progress on the DBA board.Dec 11 2017, 7:23 AM

Change 397295 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1034

https://gerrit.wikimedia.org/r/397295

Change 397295 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1034

https://gerrit.wikimedia.org/r/397295

Mentioned in SAL (#wikimedia-operations) [2017-12-11T07:49:47Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1034 - T182556 (duration: 00m 45s)

Change 398046 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1034 from config

https://gerrit.wikimedia.org/r/398046

Change 398046 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1034 from config

https://gerrit.wikimedia.org/r/398046

Mentioned in SAL (#wikimedia-operations) [2017-12-13T15:42:15Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1034 from config - T182556 (duration: 01m 09s)

Mentioned in SAL (#wikimedia-operations) [2017-12-13T15:43:34Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1034 from config - T182556 (duration: 01m 07s)

Marostegui updated the task description. (Show Details)Dec 13 2017, 3:47 PM
Marostegui updated the task description. (Show Details)

Change 398425 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove db1034

https://gerrit.wikimedia.org/r/398425

Change 398426 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s7.hosts: Remove db1034

https://gerrit.wikimedia.org/r/398426

Change 398425 merged by Marostegui:
[operations/puppet@production] mariadb: Remove db1034

https://gerrit.wikimedia.org/r/398425

Change 398427 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1034.yaml: Remove file

https://gerrit.wikimedia.org/r/398427

Change 398426 merged by jenkins-bot:
[operations/software@master] s7.hosts: Remove db1034

https://gerrit.wikimedia.org/r/398426

Change 398427 merged by Marostegui:
[operations/puppet@production] db1034.yaml: Remove file

https://gerrit.wikimedia.org/r/398427

Mentioned in SAL (#wikimedia-operations) [2017-12-15T08:26:13Z] <marostegui> Remove db1034 from tendril - T182556

Marostegui updated the task description. (Show Details)Dec 15 2017, 8:38 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-15T08:38:44Z] <marostegui> Stop MySQL on db1034 to decommission it - T182556

Marostegui moved this task from In progress to Done on the DBA board.
Marostegui added a subscriber: Cmjohnson.

@Cmjohnson this host is fully ready to be decommissioned

Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Jan 2 2018, 4:09 PM
RobH claimed this task.Feb 9 2018, 10:29 PM

Change 428837 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Removing db1034 from site.pp (decom)

https://gerrit.wikimedia.org/r/428837

Change 428837 merged by Cmjohnson:
[operations/puppet@production] Removing db1034 from site.pp (decom)

https://gerrit.wikimedia.org/r/428837

Change 428839 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removing dns entries db1029|31|34

https://gerrit.wikimedia.org/r/428839

Change 428839 merged by Cmjohnson:
[operations/dns@master] Removing dns entries db1029|31|34

https://gerrit.wikimedia.org/r/428839

Cmjohnson updated the task description. (Show Details)Apr 24 2018, 9:30 PM
Cmjohnson moved this task from Decommission to High Priority Task on the ops-eqiad board.
Cmjohnson moved this task from High Priority Task to Up next on the ops-eqiad board.
RobH reassigned this task from RobH to Cmjohnson.May 1 2018, 6:02 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH added a subscriber: RobH.
Cmjohnson closed this task as Resolved.May 7 2018, 5:14 PM
Cmjohnson updated the task description. (Show Details)