Page MenuHomePhabricator
Create Task
Maniphest T181378

Decommission db1021
Closed, ResolvedPublic
Actions

Description

db1021's data has been copied over db1053. So it is ready to be decommissioned

  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/393524/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port & change switch port label to asset tag
  • - remove production dns entries & remove hostname entries in mgmt dns
  • - puppet node clean, puppet node deactivate, salt key removed

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - remove hostname label, remove hostname from visible label field in racktables (by onsite)
  • - system added back to spares tracking (by onsite)

Details

ProjectBranchLines +/-Subject
operations/softwaremaster+0 -1s2.hosts: Remove db1021
operations/puppetproduction+6 -9mariadb: Prepare to decommission db1021
operations/mediawiki-configmaster+0 -3db-eqiad,db-codfw.php: Remove db1021
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT134476 Decommission old coredb machines (<=db1050)
 ResolvedCmjohnsonT181378 Decommission db1021

Event Timeline

• Marostegui created this task.Nov 27 2017, 6:45 AM
gerritbot added a subscriber: gerritbot.Nov 27 2017, 6:47 AM

Change 393524 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1021

https://gerrit.wikimedia.org/r/393524

gerritbot added a project: Patch-For-Review.Nov 27 2017, 6:47 AM
• Marostegui moved this task from Triage to In progress on the DBA board.Nov 27 2017, 6:48 AM
gerritbot added a comment.Nov 27 2017, 6:50 AM

Change 393524 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1021

https://gerrit.wikimedia.org/r/393524

Stashbot added a subscriber: Stashbot.Nov 27 2017, 6:52 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-27T06:52:00Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1021 from the config as it will be decommissioned - T181378 (duration: 00m 45s)

Stashbot added a comment.Nov 27 2017, 6:53 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-27T06:53:03Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1021 from the config as it will be decommissioned - T181378 (duration: 00m 44s)

gerritbot added a comment.Nov 27 2017, 6:54 AM

Change 393525 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Prepare to decommission db1021

https://gerrit.wikimedia.org/r/393525

gerritbot added a comment.Nov 27 2017, 7:03 AM

Change 393525 merged by Marostegui:
[operations/puppet@production] mariadb: Prepare to decommission db1021

https://gerrit.wikimedia.org/r/393525

gerritbot added a comment.Nov 27 2017, 7:05 AM

Change 393528 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s2.hosts: Remove db1021

https://gerrit.wikimedia.org/r/393528

gerritbot added a comment.Nov 27 2017, 7:06 AM

Change 393528 merged by jenkins-bot:
[operations/software@master] s2.hosts: Remove db1021

https://gerrit.wikimedia.org/r/393528

• Marostegui updated the task description. (Show Details)Nov 27 2017, 7:09 AM
• Marostegui updated the task description. (Show Details)
Stashbot added a comment.Nov 27 2017, 7:10 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-27T07:10:41Z] <marostegui> Stop MySQL on db1021 as it will be decommissioned - T181378

• Marostegui reassigned this task from • Marostegui to Cmjohnson.Nov 27 2017, 7:13 AM
• Marostegui moved this task from In progress to Done on the DBA board.

This server is now ready for @Cmjohnson to take over and fully decommission it.
Thanks!

Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.Nov 27 2017, 2:28 PM
• Marostegui mentioned this in T162699: Decomissions old s2 eqiad hosts (db1018, db1021, db1024, db1036).Dec 5 2017, 6:28 AM
RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.Dec 11 2017, 5:11 PM
Cmjohnson updated the task description. (Show Details)Dec 12 2017, 5:44 PM
MoritzMuehlenhoff added a subscriber: MoritzMuehlenhoff.Dec 15 2017, 12:09 PM

Still showing in servermon, also seems like a missing "puppet node deactivate"

Cmjohnson updated the task description. (Show Details)Dec 19 2017, 8:18 PM
Cmjohnson moved this task from High Priority Task to Blocked on the ops-eqiad board.Jan 3 2018, 7:20 PM
Cmjohnson closed this task as Resolved.Jan 10 2018, 3:49 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL