Page MenuHomePhabricator
Create Task
Maniphest T174763

Decommission db1026
Closed, ResolvedPublic
Actions

Description

db1026 is ready to be decommissioned after its data was copied to db1096

  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/375364/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):
  • Set it to spare until @Cmjohnson starts his steps: https://gerrit.wikimedia.org/r/#/c/375365/

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port & change switch port label to asset tag
  • - remove production dns entries & remove hostname entries in mgmt dns
  • - puppet node clean, puppet node deactivate, salt key removed

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - remove hostname label, remove hostname from visible label field in racktables (by onsite)
  • - system added back to spares tracking (by onsite)

Details

SubjectRepoBranchLines +/-
db1026.yaml: Remove fileoperations/puppetproduction+0 -2
s5.hosts: Remove db1026operations/softwaremaster+0 -1
mariadb: Decommission db1026operations/puppetproduction+6 -9
db-codfw,db-eqiad.php: Remove db1026operations/mediawiki-configmaster+0 -8
Customize query in gerrit

Event Timeline

Marostegui created this task.Sep 1 2017, 6:54 AM
Restricted Application added a project: SRE. · View Herald TranscriptSep 1 2017, 6:54 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui triaged this task as Medium priority.Sep 1 2017, 6:55 AM
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui updated the task description. (Show Details)Sep 1 2017, 12:07 PM
gerritbot subscribed.Sep 1 2017, 12:08 PM

Change 375364 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw,db-eqiad.php: Remove db1026

https://gerrit.wikimedia.org/r/375364

gerritbot added a project: Patch-For-Review.Sep 1 2017, 12:08 PM
gerritbot added a comment.Sep 1 2017, 12:16 PM

Change 375364 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw,db-eqiad.php: Remove db1026

https://gerrit.wikimedia.org/r/375364

Stashbot subscribed.Sep 1 2017, 12:17 PM

Mentioned in SAL (#wikimedia-operations) [2017-09-01T12:17:42Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s)

Stashbot added a comment.Sep 1 2017, 12:19 PM

Mentioned in SAL (#wikimedia-operations) [2017-09-01T12:19:51Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1026 as it will be decommissioned - T174763 (duration: 00m 43s)

Marostegui updated the task description. (Show Details)Sep 1 2017, 12:22 PM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Sep 1 2017, 12:24 PM

Change 375365 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Decommission db1026

https://gerrit.wikimedia.org/r/375365

gerritbot added a comment.Sep 1 2017, 12:36 PM

Change 375365 merged by Marostegui:
[operations/puppet@production] mariadb: Decommission db1026

https://gerrit.wikimedia.org/r/375365

gerritbot added a comment.Sep 1 2017, 12:39 PM

Change 375370 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s5.hosts: Remove db1026

https://gerrit.wikimedia.org/r/375370

gerritbot added a comment.Sep 1 2017, 1:09 PM

Change 375370 merged by jenkins-bot:
[operations/software@master] s5.hosts: Remove db1026

https://gerrit.wikimedia.org/r/375370

Marostegui updated the task description. (Show Details)Sep 1 2017, 1:10 PM
Marostegui added a subscriber: • Cmjohnson.
Stashbot added a comment.Sep 1 2017, 1:10 PM

Mentioned in SAL (#wikimedia-operations) [2017-09-01T13:10:40Z] <marostegui> Stop MySQL on db1026 as it will be decommissioned - T174763

Marostegui assigned this task to • Cmjohnson.Sep 1 2017, 1:13 PM
Marostegui moved this task from In progress to Done on the DBA board.

db1026 is now ready to be decommissioned and all the pending steps are DC Ops ones, so I am handing this over to @Cmjohnson

gerritbot added a comment.Sep 5 2017, 5:48 AM

Change 375950 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1026.yaml: Remove file

https://gerrit.wikimedia.org/r/375950

gerritbot added a comment.Sep 5 2017, 5:50 AM

Change 375950 merged by Marostegui:
[operations/puppet@production] db1026.yaml: Remove file

https://gerrit.wikimedia.org/r/375950

• Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Sep 5 2017, 7:27 PM
Marostegui added a project: hardware-requests.Nov 7 2017, 9:17 AM
• Cmjohnson moved this task from Decommission to High Priority Task on the ops-eqiad board.Nov 14 2017, 4:26 PM
• Cmjohnson updated the task description. (Show Details)Dec 12 2017, 5:45 PM
• Cmjohnson updated the task description. (Show Details)Dec 19 2017, 8:19 PM
MoritzMuehlenhoff subscribed.Dec 20 2017, 5:51 PM

This host still shows up in puppetdb, i.e. misses the deactivate step (e.g. visible in https://servermon.wikimedia.org/hosts/)

• Cmjohnson moved this task from High Priority Task to Blocked on the ops-eqiad board.Jan 3 2018, 7:20 PM
• Cmjohnson closed this task as Resolved.Jan 10 2018, 3:48 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL