Page MenuHomePhabricator
Create Task
Maniphest T187768

Decommission db2030
Closed, ResolvedPublic
Actions

Description

Tracking task to decommission the already unusable db2030.

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host - cannot be done due to read only filesystem
  • - remove all remaining puppet references (include role::spare) https://gerrit.wikimedia.org/r/#/c/419256/
  • - power down host (done on 2018-03-13 by @RobH via idrac as os is read only and non-responsive)
  • - disable switch port (done on 2018-03-13 by @RobH)
  • - note swtich port on task - asw-b-codfw:ge-6/0/13 (done on 2018-03-13 by @RobH)
  • - remove production dns entries & remove hostname entries in mgmt dns https://gerrit.wikimedia.org/r/#/c/419257/ (done on 2018-03-13 by @RobH)
  • - puppet node clean, puppet node deactivate (done on 2018-03-13 by @RobH)

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - system unracked and decommissioned (by onsite), update racktables with result
  • - switch port configration removed from switch once system is unracked
  • - add system to decommission tracking google sheet
  • - mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
DNS: Remove mgmt DNS for db2030operations/dnsmaster+1 -4
decom db2030 production dns entriesoperations/dnsmaster+0 -2
decom db2030operations/puppetproduction+0 -5
db-eqiad,db-codfw.php: Remove db2030 from configoperations/mediawiki-configmaster+0 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui triaged this task as Medium priority.Feb 20 2018, 10:04 AM
Marostegui created this task.
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui added a parent task: T176243: Decommission database hosts <= db2031 (tracking).
gerritbot subscribed.Feb 20 2018, 10:06 AM

Change 412868 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2030 from config

https://gerrit.wikimedia.org/r/412868

gerritbot added a project: Patch-For-Review.Feb 20 2018, 10:06 AM
gerritbot added a comment.Feb 20 2018, 10:14 AM

Change 412868 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db2030 from config

https://gerrit.wikimedia.org/r/412868

Stashbot subscribed.Feb 20 2018, 10:18 AM

Mentioned in SAL (#wikimedia-operations) [2018-02-20T10:18:15Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db2030 from config - T187768 (duration: 00m 56s)

Stashbot added a comment.Feb 20 2018, 10:20 AM

Mentioned in SAL (#wikimedia-operations) [2018-02-20T10:20:25Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db2030 from config - T187768 (duration: 00m 55s)

Marostegui updated the task description. (Show Details)Feb 20 2018, 10:20 AM
Marostegui updated the task description. (Show Details)Feb 20 2018, 11:29 AM
Marostegui updated the task description. (Show Details)Feb 21 2018, 7:00 AM

This host is now set to spare, but as puppet cannot run (FS is corrupted), the new role will never get applied :-)

Stashbot added a comment.Feb 21 2018, 10:26 AM

Mentioned in SAL (#wikimedia-operations) [2018-02-21T10:26:19Z] <marostegui> Remove db2030 from tendril - T187768

Marostegui updated the task description. (Show Details)Feb 21 2018, 10:34 AM
Marostegui reassigned this task from Marostegui to RobH.Feb 21 2018, 10:37 AM
Marostegui added projects: ops-codfw, hardware-requests.
Marostegui moved this task from In progress to Done on the DBA board.
Marostegui added a subscriber: RobH.

Assigning it directly to @RobH so he can finish up with this (please let me know if you prefer another way of letting you know that this is ready for DC Ops to move forward).

Note that the host is on spare role but the server is totally broken and the file system is on read only mode (T187722#3984136) , so you won't be able to run/disable puppet, and probably not even able to poweroff the host without doing it via the ILO.
Let me know if you need something else from my side.

Restricted Application added a project: SRE. · View Herald TranscriptFeb 21 2018, 10:37 AM
RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.Mar 5 2018, 7:54 PM
Marostegui mentioned this in T176243: Decommission database hosts <= db2031 (tracking).Mar 13 2018, 9:37 AM
RobH removed a project: Patch-For-Review.Mar 13 2018, 6:50 PM
RobH updated the task description. (Show Details)
gerritbot added a comment.Mar 13 2018, 6:50 PM

Change 419256 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom db2030

https://gerrit.wikimedia.org/r/419256

gerritbot added a project: Patch-For-Review.Mar 13 2018, 6:50 PM

Change 419256 merged by RobH:
[operations/puppet@production] decom db2030

https://gerrit.wikimedia.org/r/419256

gerritbot added a comment.Mar 13 2018, 6:54 PM

Change 419257 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom db2030 production dns entries

https://gerrit.wikimedia.org/r/419257

gerritbot added a comment.Mar 13 2018, 6:54 PM

Change 419257 merged by RobH:
[operations/dns@master] decom db2030 production dns entries

https://gerrit.wikimedia.org/r/419257

RobH reassigned this task from RobH to Papaul.Mar 13 2018, 6:58 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH added a subscriber: Papaul.

@Papaul: ready for onsite disk wipe

Papaul added a comment.Mar 13 2018, 7:26 PM

@RobH thanks

RobH moved this task from Backlog to Decommission on the ops-codfw board.Mar 15 2018, 5:13 PM
Papaul added a comment.Mar 28 2018, 3:06 PM

switch port information
asw-b6-codfw ge-6/0/13

Papaul updated the task description. (Show Details)Mar 28 2018, 3:07 PM
Papaul updated the task description. (Show Details)Mar 28 2018, 3:19 PM
gerritbot added a comment.Apr 17 2018, 3:30 PM

Change 427153 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for db2030

https://gerrit.wikimedia.org/r/427153

gerritbot added a project: Patch-For-Review.Apr 17 2018, 3:30 PM
Papaul updated the task description. (Show Details)Apr 17 2018, 3:31 PM
gerritbot added a comment.Apr 17 2018, 3:33 PM

Change 427153 merged by Marostegui:
[operations/dns@master] DNS: Remove mgmt DNS for db2030

https://gerrit.wikimedia.org/r/427153

Papaul reassigned this task from Papaul to RobH.Apr 17 2018, 3:50 PM

@RobH everything done on my side only switch port left.

Thanks

Marostegui added a subscriber: ayounsi.Apr 17 2018, 3:51 PM

Adding @ayounsi as @RobH is away.

ayounsi closed this task as Resolved.Apr 17 2018, 4:36 PM
ayounsi updated the task description. (Show Details)

Switch port cleaned up.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits