Page MenuHomePhabricator
Create Task
Maniphest T177911

Decommission db1038
Closed, ResolvedPublic
Actions

Description

db1038's data has been copied over db1072

  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/389670/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):
  • Set to spare: https://gerrit.wikimedia.org/r/#/c/389672/

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port & change switch port label to asset tag
  • - remove production dns entries & remove hostname entries in mgmt dns
  • - puppet node clean, puppet node deactivate, salt key removed

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - remove hostname label, remove hostname from visible label field in racktables (by onsite)
  • - system added back to decom rack (by onsite)

Details

SubjectRepoBranchLines +/-
mariadb: Get ready to decommission db1038operations/puppetproduction+7 -9
s3.hosts: Remove db1038operations/softwaremaster+0 -1
db-codfw,db-eqiad.php: Remove db1038 from configoperations/mediawiki-configmaster+0 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Marostegui created this task.Oct 11 2017, 8:15 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 11 2017, 8:15 AM
• Marostegui triaged this task as Medium priority.Oct 11 2017, 8:16 AM
• Marostegui moved this task from Triage to In progress on the DBA board.

db1038 can now be decommissioned.
db1072 has been pooled as vslow and dump service on s3 with db1038's data
Let's not touch it until Monday though, just to make sure db1072 works fine.

Next week I will remove db1038 from all the config and add ops-eqiad to this task.

• Marostegui added a parent task: T134476: Decommission old coredb machines (<=db1050).Oct 11 2017, 8:17 AM
• Marostegui added a parent task: T164488: Run pt-table-checksum on s3.Oct 11 2017, 8:53 AM
• Marostegui mentioned this in T148078: Decommission db1015, db1035, db1044 and db1038.Oct 26 2017, 10:25 AM
gerritbot subscribed.Nov 7 2017, 8:57 AM

Change 389670 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw,db-eqiad.php: Remove db1038 from config

https://gerrit.wikimedia.org/r/389670

gerritbot added a project: Patch-For-Review.Nov 7 2017, 8:57 AM

Change 389671 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s3.hosts: Remove db1038

https://gerrit.wikimedia.org/r/389671

gerritbot added a comment.Nov 7 2017, 9:00 AM

Change 389670 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw,db-eqiad.php: Remove db1038 from config

https://gerrit.wikimedia.org/r/389670

gerritbot added a comment.Nov 7 2017, 9:01 AM

Change 389671 merged by jenkins-bot:
[operations/software@master] s3.hosts: Remove db1038

https://gerrit.wikimedia.org/r/389671

Stashbot subscribed.Nov 7 2017, 9:02 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-07T09:02:03Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1038 from config - T177911 (duration: 00m 47s)

Stashbot added a comment.Nov 7 2017, 9:02 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-07T09:02:56Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1038 from config - T177911 (duration: 00m 45s)

gerritbot added a comment.Nov 7 2017, 9:05 AM

Change 389672 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Get ready to decommission db1038

https://gerrit.wikimedia.org/r/389672

gerritbot added a comment.Nov 7 2017, 9:12 AM

Change 389672 merged by Marostegui:
[operations/puppet@production] mariadb: Get ready to decommission db1038

https://gerrit.wikimedia.org/r/389672

• Marostegui updated the task description. (Show Details)Nov 7 2017, 9:13 AM
Stashbot added a comment.Nov 7 2017, 9:13 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-07T09:13:59Z] <marostegui> Stop MySQL on db1038 - host to be decommissioned - T177911

• Marostegui moved this task from In progress to Done on the DBA board.Nov 7 2017, 9:15 AM
• Marostegui added projects: ops-eqiad, hardware-requests.
• Marostegui added a subscriber: • Cmjohnson.

This host is fully ready to be decommissioned by @Cmjohnson

Restricted Application added a project: SRE. · View Herald TranscriptNov 7 2017, 9:15 AM
• Marostegui reassigned this task from • Marostegui to • Cmjohnson.Nov 7 2017, 9:15 AM
• Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Nov 7 2017, 3:33 PM
• Cmjohnson moved this task from Decommission to High Priority Task on the ops-eqiad board.
• Cmjohnson updated the task description. (Show Details)Nov 8 2017, 6:52 PM
• Cmjohnson updated the task description. (Show Details)Nov 8 2017, 8:37 PM
• Cmjohnson closed this task as Resolved.Nov 14 2017, 4:25 PM

Wiped, racktables updated

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL