Page MenuHomePhabricator

Decommission db2030
Closed, ResolvedPublic

Description

Tracking task to decommission the already unusable db2030.

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host - cannot be done due to read only filesystem
  • - remove all remaining puppet references (include role::spare) https://gerrit.wikimedia.org/r/#/c/419256/
  • - power down host (done on 2018-03-13 by @RobH via idrac as os is read only and non-responsive)
  • - disable switch port (done on 2018-03-13 by @RobH)
  • - note swtich port on task - asw-b-codfw:ge-6/0/13 (done on 2018-03-13 by @RobH)
  • - remove production dns entries & remove hostname entries in mgmt dns https://gerrit.wikimedia.org/r/#/c/419257/ (done on 2018-03-13 by @RobH)
  • - puppet node clean, puppet node deactivate (done on 2018-03-13 by @RobH)

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - system unracked and decommissioned (by onsite), update racktables with result
  • - switch port configration removed from switch once system is unracked
  • - add system to decommission tracking google sheet
  • - mgmt dns entries removed.

Details

Related Gerrit Patches:
operations/dns : masterDNS: Remove mgmt DNS for db2030
operations/dns : masterdecom db2030 production dns entries
operations/puppet : productiondecom db2030
operations/mediawiki-config : masterdb-eqiad,db-codfw.php: Remove db2030 from config

Related Objects

Event Timeline

Marostegui triaged this task as Medium priority.Feb 20 2018, 10:04 AM
Marostegui created this task.
Marostegui moved this task from Triage to In progress on the DBA board.

Change 412868 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2030 from config

https://gerrit.wikimedia.org/r/412868

Change 412868 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2030 from config

https://gerrit.wikimedia.org/r/412868

Mentioned in SAL (#wikimedia-operations) [2018-02-20T10:18:15Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db2030 from config - T187768 (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2018-02-20T10:20:25Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db2030 from config - T187768 (duration: 00m 55s)

Marostegui updated the task description. (Show Details)Feb 20 2018, 10:20 AM
Marostegui updated the task description. (Show Details)Feb 20 2018, 11:29 AM
Marostegui updated the task description. (Show Details)Feb 21 2018, 7:00 AM

This host is now set to spare, but as puppet cannot run (FS is corrupted), the new role will never get applied :-)

Mentioned in SAL (#wikimedia-operations) [2018-02-21T10:26:19Z] <marostegui> Remove db2030 from tendril - T187768

Marostegui updated the task description. (Show Details)Feb 21 2018, 10:34 AM
Marostegui reassigned this task from Marostegui to RobH.Feb 21 2018, 10:37 AM
Marostegui moved this task from In progress to Done on the DBA board.
Marostegui added a subscriber: RobH.

Assigning it directly to @RobH so he can finish up with this (please let me know if you prefer another way of letting you know that this is ready for DC Ops to move forward).

Note that the host is on spare role but the server is totally broken and the file system is on read only mode (T187722#3984136) , so you won't be able to run/disable puppet, and probably not even able to poweroff the host without doing it via the ILO.
Let me know if you need something else from my side.

Restricted Application added a project: Operations. · View Herald TranscriptFeb 21 2018, 10:37 AM
RobH updated the task description. (Show Details)

Change 419256 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom db2030

https://gerrit.wikimedia.org/r/419256

Change 419256 merged by RobH:
[operations/puppet@production] decom db2030

https://gerrit.wikimedia.org/r/419256

Change 419257 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom db2030 production dns entries

https://gerrit.wikimedia.org/r/419257

Change 419257 merged by RobH:
[operations/dns@master] decom db2030 production dns entries

https://gerrit.wikimedia.org/r/419257

RobH reassigned this task from RobH to Papaul.Mar 13 2018, 6:58 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH added a subscriber: Papaul.

@Papaul: ready for onsite disk wipe

RobH moved this task from Backlog to Decommission on the ops-codfw board.Mar 15 2018, 5:13 PM

switch port information
asw-b6-codfw ge-6/0/13

Papaul updated the task description. (Show Details)Mar 28 2018, 3:07 PM
Papaul updated the task description. (Show Details)Mar 28 2018, 3:19 PM

Change 427153 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for db2030

https://gerrit.wikimedia.org/r/427153

Papaul updated the task description. (Show Details)Apr 17 2018, 3:31 PM

Change 427153 merged by Marostegui:
[operations/dns@master] DNS: Remove mgmt DNS for db2030

https://gerrit.wikimedia.org/r/427153

Papaul reassigned this task from Papaul to RobH.Apr 17 2018, 3:50 PM

@RobH everything done on my side only switch port left.

Thanks

ayounsi closed this task as Resolved.Apr 17 2018, 4:36 PM
ayounsi updated the task description. (Show Details)

Switch port cleaned up.