Page MenuHomePhabricator
Create Task
Maniphest T182556

Decommission db1034
Closed, ResolvedPublic
Actions

Description

db1034 is ready to be decommissioned after transferring its data to the new multi-instance hosts in s7

  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/398046/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):
  • Host set to spare: https://gerrit.wikimedia.org/r/#/c/398425/

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port & change switch port label to asset tag
  • - remove production dns entries & remove hostname entries in mgmt dns
  • - puppet node clean, puppet node deactivate, salt key removed

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
Removing dns entries db1029|31|34operations/dnsmaster+0 -22
Removing db1034 from site.pp (decom)operations/puppetproduction+0 -5
db1034.yaml: Remove fileoperations/puppetproduction+0 -2
s7.hosts: Remove db1034operations/softwaremaster+0 -1
mariadb: Remove db1034operations/puppetproduction+4 -9
db-eqiad,db-codfw.php: Remove db1034 from configoperations/mediawiki-configmaster+0 -8
db-eqiad.php: Depool db1034operations/mediawiki-configmaster+6 -6
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT134476 Decommission old coredb machines (<=db1050)
 Resolved• CmjohnsonT182556 Decommission db1034

Event Timeline

Marostegui triaged this task as Medium priority.Dec 11 2017, 7:23 AM
Marostegui created this task.
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptDec 11 2017, 7:23 AM
Marostegui moved this task from Triage to In progress on the DBA board.Dec 11 2017, 7:23 AM
gerritbot subscribed.Dec 11 2017, 7:25 AM

Change 397295 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1034

https://gerrit.wikimedia.org/r/397295

gerritbot added a project: Patch-For-Review.Dec 11 2017, 7:25 AM
gerritbot added a comment.Dec 11 2017, 7:48 AM

Change 397295 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1034

https://gerrit.wikimedia.org/r/397295

Stashbot added a comment.Dec 11 2017, 7:49 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-11T07:49:47Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1034 - T182556 (duration: 00m 45s)

gerritbot added a comment.Dec 13 2017, 1:48 PM

Change 398046 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1034 from config

https://gerrit.wikimedia.org/r/398046

gerritbot added a comment.Dec 13 2017, 3:40 PM

Change 398046 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1034 from config

https://gerrit.wikimedia.org/r/398046

Stashbot added a comment.Dec 13 2017, 3:42 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-13T15:42:15Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1034 from config - T182556 (duration: 01m 09s)

Stashbot added a comment.Dec 13 2017, 3:43 PM

Mentioned in SAL (#wikimedia-operations) [2017-12-13T15:43:34Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1034 from config - T182556 (duration: 01m 07s)

Marostegui updated the task description. (Show Details)Dec 13 2017, 3:47 PM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Dec 15 2017, 8:17 AM

Change 398425 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove db1034

https://gerrit.wikimedia.org/r/398425

gerritbot added a comment.Dec 15 2017, 8:21 AM

Change 398426 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s7.hosts: Remove db1034

https://gerrit.wikimedia.org/r/398426

gerritbot added a comment.Dec 15 2017, 8:22 AM

Change 398425 merged by Marostegui:
[operations/puppet@production] mariadb: Remove db1034

https://gerrit.wikimedia.org/r/398425

gerritbot added a comment.Dec 15 2017, 8:24 AM

Change 398427 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1034.yaml: Remove file

https://gerrit.wikimedia.org/r/398427

gerritbot added a comment.Dec 15 2017, 8:25 AM

Change 398426 merged by jenkins-bot:
[operations/software@master] s7.hosts: Remove db1034

https://gerrit.wikimedia.org/r/398426

gerritbot added a comment.Dec 15 2017, 8:25 AM

Change 398427 merged by Marostegui:
[operations/puppet@production] db1034.yaml: Remove file

https://gerrit.wikimedia.org/r/398427

Stashbot added a comment.Dec 15 2017, 8:26 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-15T08:26:13Z] <marostegui> Remove db1034 from tendril - T182556

Marostegui updated the task description. (Show Details)Dec 15 2017, 8:38 AM
Stashbot added a comment.Dec 15 2017, 8:38 AM

Mentioned in SAL (#wikimedia-operations) [2017-12-15T08:38:44Z] <marostegui> Stop MySQL on db1034 to decommission it - T182556

Marostegui reassigned this task from Marostegui to • Cmjohnson.Dec 15 2017, 8:42 AM
Marostegui added projects: ops-eqiad, hardware-requests.
Marostegui moved this task from In progress to Done on the DBA board.
Marostegui added a subscriber: • Cmjohnson.

@Cmjohnson this host is fully ready to be decommissioned

RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.Dec 20 2017, 5:38 PM
• Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Jan 2 2018, 4:09 PM
RobH claimed this task.Feb 9 2018, 10:29 PM
Marostegui mentioned this in T191792: Rack and setup db1116 - db1123.Apr 9 2018, 1:20 PM
gerritbot added a comment.Apr 24 2018, 9:19 PM

Change 428837 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/puppet@production] Removing db1034 from site.pp (decom)

https://gerrit.wikimedia.org/r/428837

gerritbot added a comment.Apr 24 2018, 9:22 PM

Change 428837 merged by Cmjohnson:
[operations/puppet@production] Removing db1034 from site.pp (decom)

https://gerrit.wikimedia.org/r/428837

gerritbot added a comment.Apr 24 2018, 9:29 PM

Change 428839 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removing dns entries db1029|31|34

https://gerrit.wikimedia.org/r/428839

gerritbot added a comment.Apr 24 2018, 9:29 PM

Change 428839 merged by Cmjohnson:
[operations/dns@master] Removing dns entries db1029|31|34

https://gerrit.wikimedia.org/r/428839

• Cmjohnson updated the task description. (Show Details)Apr 24 2018, 9:30 PM
• Cmjohnson moved this task from Decommission to High Priority Task on the ops-eqiad board.
• Cmjohnson moved this task from High Priority Task to Up next on the ops-eqiad board.
RobH reassigned this task from RobH to • Cmjohnson.May 1 2018, 6:02 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH subscribed.
RobH edited projects, added decommission-hardware; removed hardware-requests.May 3 2018, 10:58 PM
• Cmjohnson closed this task as Resolved.May 7 2018, 5:14 PM
• Cmjohnson updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL