Page MenuHomePhabricator
Create Task
Maniphest T173570

Decommission db1015
Closed, ResolvedPublic
Actions

Description

db1015 is ready to be decommissioned:

  • - all system services confirmed offline from production use: Removed from mediawiki-config: https://gerrit.wikimedia.org/r/#/c/372816/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/heira/dsh config removed
  • - remove site.pp (replace with role::spare if system isn't shut down immediately during this process.):
  • Host set to spare until @Cmjohnson removes it forever. https://gerrit.wikimedia.org/r/#/c/372818/

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port & change switch port label to asset tag
  • - remove production dns entries & remove hostname entries in mgmt dns
  • - puppet node clean, puppet node deactivate, salt key removed

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - remove hostname label, remove hostname from visible label field in racktables (by onsite)
  • - system added back to spares tracking (by onsite)

Details

SubjectRepoBranchLines +/-
mariadb: Remove yaml files from db1015 and db1041operations/puppetproduction+0 -4
s3.hosts: Remove db1015operations/softwaremaster+0 -1
mariadb: Remove db1015operations/puppetproduction+6 -9
db-eqiad,db-codfw.php: Remove db1015operations/mediawiki-configmaster+0 -3
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT134476 Decommission old coredb machines (<=db1050)
 Resolved• CmjohnsonT173570 Decommission db1015

Event Timeline

Marostegui created this task.Aug 18 2017, 12:50 PM
Restricted Application added a project: SRE. · View Herald TranscriptAug 18 2017, 12:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui moved this task from Triage to Pending comment on the DBA board.Aug 18 2017, 12:50 PM
Marostegui updated the task description. (Show Details)
Peachey88 added a project: hardware-requests.Aug 19 2017, 4:09 AM
Peachey88 moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.
gerritbot subscribed.Aug 21 2017, 8:22 AM

Change 372816 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1015

https://gerrit.wikimedia.org/r/372816

gerritbot added a project: Patch-For-Review.Aug 21 2017, 8:22 AM

Change 372817 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s3.hosts: Remove db1015

https://gerrit.wikimedia.org/r/372817

gerritbot added a comment.Aug 21 2017, 8:26 AM

Change 372818 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove db1015

https://gerrit.wikimedia.org/r/372818

gerritbot added a comment.Aug 21 2017, 9:14 AM

Change 372816 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1015

https://gerrit.wikimedia.org/r/372816

Stashbot subscribed.Aug 21 2017, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2017-08-21T09:15:52Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1015 - T173570 (duration: 00m 44s)

Stashbot added a comment.Aug 21 2017, 9:17 AM

Mentioned in SAL (#wikimedia-operations) [2017-08-21T09:17:16Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1015 - T173570 (duration: 00m 44s)

gerritbot added a comment.Aug 21 2017, 12:27 PM

Change 372818 merged by Marostegui:
[operations/puppet@production] mariadb: Remove db1015

https://gerrit.wikimedia.org/r/372818

gerritbot added a comment.Aug 21 2017, 12:35 PM

Change 372817 merged by jenkins-bot:
[operations/software@master] s3.hosts: Remove db1015

https://gerrit.wikimedia.org/r/372817

Stashbot added a comment.Aug 21 2017, 12:35 PM

Mentioned in SAL (#wikimedia-operations) [2017-08-21T12:35:47Z] <marostegui> Stop MySQL on db1015 to decommission it - T173570

Marostegui assigned this task to • Cmjohnson.Aug 21 2017, 12:38 PM
Marostegui updated the task description. (Show Details)
Marostegui moved this task from Pending comment to Blocked external/Not db team on the DBA board.

This host is now ready for the remaining steps from @Cmjohnson

• Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Aug 21 2017, 10:16 PM
Marostegui moved this task from Blocked external/Not db team to Done on the DBA board.Aug 22 2017, 10:25 AM
gerritbot added a comment.Aug 24 2017, 12:41 PM

Change 373534 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove yaml files from db1015 and db1041

https://gerrit.wikimedia.org/r/373534

gerritbot added a comment.Aug 24 2017, 12:47 PM

Change 373534 merged by Marostegui:
[operations/puppet@production] mariadb: Remove yaml files from db1015 and db1041

https://gerrit.wikimedia.org/r/373534

Marostegui mentioned this in T148078: Decommission db1015, db1035, db1044 and db1038.Sep 25 2017, 12:24 PM
Marostegui mentioned this in T177405: rack and setup db1107 and db1108.Oct 4 2017, 5:35 PM
Stashbot added a comment.Oct 23 2017, 10:28 AM

Mentioned in SAL (#wikimedia-operations) [2017-10-23T10:28:03Z] <marostegui> Remove /srv from db1015 as it has been stopped for weeks now and will be decommissioned (and it is alerting low on disk space) - T173570

• Cmjohnson added a comment.Nov 8 2017, 8:57 PM

@Marostegui during my decom checks I found db1015 in this file. Should a replacement be identified?

modules/admin/files/enforce-users-groups.sh

Marostegui added a comment.Nov 9 2017, 6:42 AM
In T173570#3745891, @Cmjohnson wrote:

@Marostegui during my decom checks I found db1015 in this file. Should a replacement be identified?

modules/admin/files/enforce-users-groups.sh

Hi Chris,

No worries about that, you can safely decommission this host.
Thanks for the heads up!

• Cmjohnson moved this task from Decommission to High Priority Task on the ops-eqiad board.Nov 14 2017, 4:26 PM
• Cmjohnson updated the task description. (Show Details)Dec 12 2017, 5:42 PM
• Cmjohnson updated the task description. (Show Details)Dec 19 2017, 8:18 PM
• MoritzMuehlenhoff subscribed.Dec 20 2017, 5:50 PM

This host still shows up in puppetdb, i.e. misses the deactivate step (e.g. visible in https://servermon.wikimedia.org/hosts/)

• Cmjohnson moved this task from High Priority Task to Blocked on the ops-eqiad board.Jan 3 2018, 7:20 PM
• Cmjohnson closed this task as Resolved.Jan 10 2018, 3:47 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits