Page MenuHomePhabricator
Create Task
Maniphest T184262

Decommission db1039
Closed, ResolvedPublic
Actions

Description

db1039 is ready to be decommissioned after being finished with data checksums.

  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/402765/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):
  • Host set to spare: https://gerrit.wikimedia.org/r/#/c/402766/1

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port & change switch port label to asset tag
  • - remove production dns entries & remove hostname entries in mgmt dns
  • - puppet node clean, puppet node deactivate, salt key removed

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - remove hostname label, remove hostname from visible label field in racktables (by onsite)
  • - system added back to spares tracking (by onsite)

Details

SubjectRepoBranchLines +/-
Removing dns entries for db1039operations/dnsmaster+0 -6
Removing db1039 site.pp entryoperations/puppetproduction+0 -5
s7.hosts: Remove db1039operations/softwaremaster+0 -1
mariadb: Remove db1039operations/puppetproduction+10 -13
db-eqiad,db-codfw.php: Remove db1039 from configoperations/mediawiki-configmaster+0 -3
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT134476 Decommission old coredb machines (<=db1050)
 ResolvedRobHT184262 Decommission db1039

Event Timeline

• Marostegui triaged this task as Medium priority.Jan 5 2018, 10:15 AM
• Marostegui created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 5 2018, 10:15 AM
• Marostegui moved this task from Triage to Pending comment on the DBA board.Jan 5 2018, 10:16 AM

I will start with it on Monday

• Marostegui moved this task from Pending comment to In progress on the DBA board.Jan 8 2018, 6:55 AM
gerritbot subscribed.Jan 8 2018, 6:56 AM

Change 402765 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1039 from config

https://gerrit.wikimedia.org/r/402765

gerritbot added a project: Patch-For-Review.Jan 8 2018, 6:56 AM
gerritbot added a comment.Jan 8 2018, 7:00 AM

Change 402766 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove db1039

https://gerrit.wikimedia.org/r/402766

gerritbot added a comment.Jan 8 2018, 7:03 AM

Change 402765 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1039 from config

https://gerrit.wikimedia.org/r/402765

Stashbot subscribed.Jan 8 2018, 7:04 AM

Mentioned in SAL (#wikimedia-operations) [2018-01-08T07:04:32Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1039 as it will be decommissioned - T184262 (duration: 00m 50s)

Stashbot added a comment.Jan 8 2018, 7:05 AM

Mentioned in SAL (#wikimedia-operations) [2018-01-08T07:05:29Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1039 as it will be decommissioned - T184262 (duration: 00m 50s)

gerritbot added a comment.Jan 8 2018, 7:13 AM

Change 402766 merged by Marostegui:
[operations/puppet@production] mariadb: Remove db1039

https://gerrit.wikimedia.org/r/402766

Stashbot added a comment.Jan 8 2018, 7:17 AM

Mentioned in SAL (#wikimedia-operations) [2018-01-08T07:17:10Z] <marostegui> Remove db1039 from tendril - T184262

Stashbot added a comment.Jan 8 2018, 7:24 AM

Mentioned in SAL (#wikimedia-operations) [2018-01-08T07:24:13Z] <marostegui> Stop MySQL on db1039 for decommission - T184262

gerritbot added a comment.Jan 8 2018, 7:24 AM

Change 402768 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s7.hosts: Remove db1039

https://gerrit.wikimedia.org/r/402768

• Marostegui updated the task description. (Show Details)Jan 8 2018, 7:26 AM
gerritbot added a comment.Jan 8 2018, 7:26 AM

Change 402768 merged by jenkins-bot:
[operations/software@master] s7.hosts: Remove db1039

https://gerrit.wikimedia.org/r/402768

• Marostegui reassigned this task from • Marostegui to • Cmjohnson.Jan 8 2018, 7:27 AM
• Marostegui moved this task from In progress to Done on the DBA board.
• Marostegui added a project: ops-eqiad.
• Marostegui added a subscriber: • Cmjohnson.

db1039 is now ready to be decommissioned by @Cmjohnson

Restricted Application added a project: SRE. · View Herald TranscriptJan 8 2018, 7:27 AM
• Marostegui added a project: hardware-requests.Jan 8 2018, 8:10 AM
• Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Jan 9 2018, 3:20 PM
RobH claimed this task.Feb 9 2018, 10:27 PM
RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.
• Marostegui mentioned this in T191792: Rack and setup db1116 - db1123.Apr 9 2018, 1:20 PM
gerritbot added a comment.Apr 25 2018, 2:49 PM

Change 428932 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Removing db1039 site.pp entry

https://gerrit.wikimedia.org/r/428932

gerritbot added a comment.Apr 25 2018, 2:50 PM

Change 428932 merged by Cmjohnson:
[operations/puppet@production] Removing db1039 site.pp entry

https://gerrit.wikimedia.org/r/428932

gerritbot added a comment.Apr 30 2018, 2:37 PM

Change 429814 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removing dns entries for db1039

https://gerrit.wikimedia.org/r/429814

gerritbot added a comment.Apr 30 2018, 2:46 PM

Change 429814 merged by Cmjohnson:
[operations/dns@master] Removing dns entries for db1039

https://gerrit.wikimedia.org/r/429814

• Cmjohnson updated the task description. (Show Details)Apr 30 2018, 2:46 PM
• Cmjohnson moved this task from Decommission to High Priority Task on the ops-eqiad board.Apr 30 2018, 2:51 PM
RobH edited projects, added decommission-hardware; removed hardware-requests.May 3 2018, 10:59 PM
• Cmjohnson closed this task as Resolved.May 7 2018, 5:13 PM
• Cmjohnson updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits