Page MenuHomePhabricator
Create Task
Maniphest T187542

Decommission db1043
Closed, ResolvedPublic
Actions

Description

db1043 has been substituted by db1053. After one week, db1053 is working properly, we are free to decommission it.

Decommission Checklist

START NON-INTERRUPPTABLE STEPS - please assign to @RobH for the non-interrupt steps

  • - disable puppet on host
  • - power down host
  • - disable switch port - CANNOT DO AS SYSTEM IS NOT LABELED ON SWITCH also the mac doesn't show up in the switching table
  • - switch port assignment noted on this task (for later removal) cannot do, but needs to be traced and set to disabled by @Cmjohnson
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - onsite trace and disable the network port (already has been unplugged)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
Removal of mgmt dns db1043operations/dnsmaster+0 -5
decom db1043operations/puppetproduction+0 -12
decom db1043operations/dnsmaster+0 -3
Mariadb: Schedule db1043 and db2012 for decommissionoperations/puppetproduction+10 -12
Customize query in gerrit

Related Objects
Search...

Event Timeline

jcrespo triaged this task as Medium priority.Feb 16 2018, 12:53 PM
jcrespo created this task.
gerritbot subscribed.Feb 16 2018, 1:11 PM

Change 411229 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] Mariadb: Schedule db1043 and db2012 for decommission

https://gerrit.wikimedia.org/r/411229

gerritbot added a project: Patch-For-Review.Feb 16 2018, 1:11 PM
gerritbot added a comment.Feb 16 2018, 1:40 PM

Change 411229 merged by Jcrespo:
[operations/puppet@production] Mariadb: Schedule db1043 and db2012 for decommission

https://gerrit.wikimedia.org/r/411229

• Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Feb 16 2018, 3:47 PM
RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.Feb 16 2018, 5:34 PM
RobH added a project: DBA.Feb 16 2018, 5:37 PM
RobH updated the task description. (Show Details)

Since this is pending the DBA team's work on stating the new host is online, I've appended in the DBA flag. Once the DBA team work is done (their sign off its ok to continue and the replacement is working & the decom steps flagged for them due to service interactions), this can be assigned to @RobH and the DBA team flag removed from the task.

(If anything of the above is incorrect in the use of the DBA tag, my apologies! It seemed the best way to ensure this stays on radar without directly pinging anyone.)

jcrespo added a comment.Feb 17 2018, 4:37 PM

The usage of the tafs is ok.

Note the substitution host is already online and in production, and the old hosts set as spare. What we wanted to to wait, e.g. 7 days to "assign" it to you (but you had and advance heads up, if that helps with something.) in the 1 in a million case that the data copied was not done properly twice, backups were unusable, and new host that has been there for months also broke, and we had to put the old servers back into production. So let's say to proceed with the decommission on the 26th February or later unless you hear back from us. In theory decomm could start already except for the un-doable disk wipe.

• Marostegui moved this task from Triage to In progress on the DBA board.Feb 21 2018, 4:14 PM
jcrespo reassigned this task from jcrespo to RobH.Feb 23 2018, 5:49 PM
jcrespo removed a project: Patch-For-Review.
jcrespo updated the task description. (Show Details)
jcrespo updated the task description. (Show Details)
jcrespo moved this task from In progress to Done on the DBA board.
RobH updated the task description. (Show Details)Mar 14 2018, 5:17 PM
gerritbot added a comment.Mar 14 2018, 5:23 PM

Change 419487 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom db1043

https://gerrit.wikimedia.org/r/419487

gerritbot added a project: Patch-For-Review.Mar 14 2018, 5:23 PM
gerritbot added a comment.Mar 14 2018, 5:26 PM

Change 419488 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom db1043

https://gerrit.wikimedia.org/r/419488

RobH reassigned this task from RobH to • Cmjohnson.Mar 14 2018, 5:27 PM

Please note that the switch port for this host was not labeled & doesn't show in ethernet switching table. So @Cmjohnson went ahead and unplugged it for me, so we don't have any issues of a decom host coming back online before wipe.

When he goes to do the disk wipe, he will need to trace and disable the network port.

gerritbot added a comment.Mar 14 2018, 5:28 PM

Change 419487 merged by RobH:
[operations/dns@master] decom db1043

https://gerrit.wikimedia.org/r/419487

gerritbot added a comment.Mar 14 2018, 5:29 PM

Change 419488 merged by RobH:
[operations/puppet@production] decom db1043

https://gerrit.wikimedia.org/r/419488

RobH removed a project: Patch-For-Review.Mar 14 2018, 5:30 PM
RobH updated the task description. (Show Details)
• Cmjohnson moved this task from Decommission to High Priority Task on the ops-eqiad board.Mar 22 2018, 7:28 PM
gerritbot added a comment.Mar 23 2018, 4:18 PM

Change 421560 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removal of mgmt dns db1043

https://gerrit.wikimedia.org/r/421560

gerritbot added a project: Patch-For-Review.Mar 23 2018, 4:18 PM

Change 421560 merged by Cmjohnson:
[operations/dns@master] Removal of mgmt dns db1043

https://gerrit.wikimedia.org/r/421560

• Cmjohnson closed this task as Resolved.Mar 23 2018, 4:21 PM
• Cmjohnson updated the task description. (Show Details)
• Marostegui added a parent task: T134476: Decommission old coredb machines (<=db1050).Mar 23 2018, 4:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL