Page MenuHomePhabricator
Create Task
Maniphest T178162

Decommission db1050
Closed, ResolvedPublic
Actions

Description

db1050 can be decommissioned

  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/384453/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):
  • Set to spare: https://gerrit.wikimedia.org/r/#/c/384455/

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port & change switch port label to asset tag
  • - remove production dns entries & remove hostname entries in mgmt dns
  • - puppet node clean, puppet node deactivate, salt key removed

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - remove hostname label, remove hostname from visible label field in racktables (by onsite)
  • - system added back to spares tracking (by onsite)

Details

SubjectRepoBranchLines +/-
mariadb: Remove db1050 from configoperations/puppetproduction+7 -11
s6.hosts: Remove db1050operations/softwaremaster+0 -1
db-eqiad,db-codfw.php: Remove db1050 from configoperations/mediawiki-configmaster+0 -3
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT134476 Decommission old coredb machines (<=db1050)
 Resolved• CmjohnsonT178162 Decommission db1050

Event Timeline

• Marostegui created this task.Oct 13 2017, 12:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 13 2017, 12:45 PM
• Marostegui triaged this task as Medium priority.Oct 13 2017, 12:45 PM
• Marostegui added a parent task: T134476: Decommission old coredb machines (<=db1050).
• Marostegui moved this task from Triage to Pending comment on the DBA board.
gerritbot subscribed.Oct 16 2017, 8:25 AM

Change 384453 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1050 from config

https://gerrit.wikimedia.org/r/384453

gerritbot added a project: Patch-For-Review.Oct 16 2017, 8:25 AM
gerritbot added a comment.Oct 16 2017, 8:29 AM

Change 384453 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1050 from config

https://gerrit.wikimedia.org/r/384453

Stashbot subscribed.Oct 16 2017, 8:30 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-16T08:30:07Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1050 from config - T178162 (duration: 00m 46s)

gerritbot added a comment.Oct 16 2017, 8:30 AM

Change 384454 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s6.hosts: Remove db1050

https://gerrit.wikimedia.org/r/384454

Stashbot added a comment.Oct 16 2017, 8:31 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-16T08:31:02Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1050 from config - T178162 (duration: 00m 46s)

gerritbot added a comment.Oct 16 2017, 8:33 AM

Change 384455 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove db1050 from config

https://gerrit.wikimedia.org/r/384455

gerritbot added a comment.Oct 16 2017, 8:33 AM

Change 384454 merged by jenkins-bot:
[operations/software@master] s6.hosts: Remove db1050

https://gerrit.wikimedia.org/r/384454

gerritbot added a comment.Oct 16 2017, 8:44 AM

Change 384455 merged by Marostegui:
[operations/puppet@production] mariadb: Remove db1050 from config

https://gerrit.wikimedia.org/r/384455

Stashbot added a comment.Oct 16 2017, 8:46 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-16T08:46:23Z] <marostegui> Remove db1050 from tendril - T178162

Stashbot added a comment.Oct 16 2017, 8:53 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-16T08:53:52Z] <marostegui> Stop MySQL on db1050 as it will be decommissioned - T178162

• Marostegui assigned this task to • Cmjohnson.Oct 16 2017, 8:54 AM
• Marostegui updated the task description. (Show Details)
• Marostegui moved this task from Pending comment to Done on the DBA board.
• Marostegui added a project: ops-eqiad.
• Marostegui added a subscriber: • Cmjohnson.

db1050 can now be decommissioned by @Cmjohnson
@Cmjohnson remember that one of the disks is failed, so it would be good to identify that one before decommissioning it, so we do not re-use that broken one in the future
Thanks!

Restricted Application added a project: SRE. · View Herald TranscriptOct 16 2017, 8:56 AM
• Marostegui updated the task description. (Show Details)Oct 16 2017, 8:56 AM
• Marostegui updated the task description. (Show Details)
• Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Oct 18 2017, 2:48 PM
• Marostegui added a project: hardware-requests.Nov 7 2017, 9:16 AM
RobH mentioned this in T179727: Degraded RAID on db1059.Nov 7 2017, 4:00 PM
jcrespo subscribed.Nov 14 2017, 3:04 PM

@Cmjohnson This would be on the top of the db decommissioning stack (which obviously, is not that urgent) so we can get rid of the non-useful alert about the bad disk.

• Marostegui added a comment.Nov 14 2017, 3:05 PM

And let's make sure we mark that bad disk as broken so it is not re-used somewhere else :-)

• Cmjohnson moved this task from Decommission to High Priority Task on the ops-eqiad board.Nov 14 2017, 4:26 PM
RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.Dec 11 2017, 5:11 PM
• Cmjohnson updated the task description. (Show Details)Dec 12 2017, 5:49 PM
• Cmjohnson updated the task description. (Show Details)Dec 19 2017, 8:20 PM
• Cmjohnson closed this task as Resolved.Jan 3 2018, 7:15 PM
• Cmjohnson updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL