Page MenuHomePhabricator
Create Task
Maniphest T216491

Decommission dbstore1002
Closed, ResolvedPublic
Actions

Description

dbstore1002 has been migrated to the new hosts (T210478) and thus can be decommissioned

Wait for the green light - dbstore1002 is still being used (for reads)

  • Analytics to confirm this host can be decommissioned

dbstore1002

Decommission Checklist

  • - all system services confirmed offline from production use - should be done by Analytics team
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration - should be done by Analytics team: Removed from config:
  • - any service group puppet/heira/dsh config removed - should be done by Analytics team
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.) - should be done by Analytics team: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/494649/

START NON-INTERRUPPTABLE STEPS - please assign to @RobH for the non-interrupt steps

  • - disable puppet on host
  • - power down host
  • - update status in netbox (inventory for decom, planned for spare)
  • - disable switch port
  • - switch port assignment noted on this task (for later removal) asw2-d-eqiad:ge-1/0/7
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate (handled by wmf-decommission-host)
  • - remove dbmonitor entries on neodymium/sarin: sudo curl -X DELETE https://debmonitor.discovery.wmnet/hosts/${HOST_FQDN} --cert /etc/debmonitor/ssl/cert.pem --key /etc/debmonitor/ssl/server.key (handled by wmf-decommission-host)

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

SubjectRepoBranchLines +/-
DNS: Remove mgmt DNS for analytic1003, dbstore1002 abd ms-be1027operations/dnsmaster+1 -12
aliases.yaml.erb: Remove dbstoreoperations/puppetproduction+1 -1
mariadb: Remove mariadb::dbstoreoperations/puppetproduction+1 -220
decom dbstore1002operations/puppetproduction+0 -4
decom dbstore1002operations/dnsmaster+1 -2
dbstore1002: Set it to spareoperations/puppetproduction+1 -7
Delete analytics-store CNAME due to dbstore1002 decomoperations/dnsmaster+0 -1
install_server: Remove dbstore1002operations/puppetproduction+0 -7
analytics-grants.sql: Remove fileoperations/puppetproduction+0 -10
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui changed the task status from Open to Stalled.Feb 19 2019, 10:43 AM
Marostegui triaged this task as Medium priority.
Marostegui created this task.
MoritzMuehlenhoff subscribed.Feb 19 2019, 10:44 AM
Marostegui added a subtask: T210478: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5].Feb 19 2019, 10:44 AM
Milimetric moved this task from Incoming to Radar on the Analytics board.Feb 21 2019, 5:46 PM
• Cmjohnson moved this task from Backlog to Decommission on the ops-eqiad board.Feb 21 2019, 6:42 PM
Marostegui mentioned this in T210478: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5].Feb 22 2019, 6:56 AM
gerritbot subscribed.Feb 22 2019, 10:02 AM

Change 492275 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] analytics-grants.sql: Remove file

https://gerrit.wikimedia.org/r/492275

gerritbot added a project: Patch-For-Review.Feb 22 2019, 10:02 AM
gerritbot added a comment.Feb 22 2019, 10:53 AM

Change 492275 merged by Marostegui:
[operations/puppet@production] analytics-grants.sql: Remove file

https://gerrit.wikimedia.org/r/492275

gerritbot added a comment.Mar 1 2019, 6:10 AM

Change 493647 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Remove dbstore1002

https://gerrit.wikimedia.org/r/493647

gerritbot added a comment.Mar 1 2019, 6:11 AM

Change 493647 merged by Marostegui:
[operations/puppet@production] install_server: Remove dbstore1002

https://gerrit.wikimedia.org/r/493647

Stashbot mentioned this in T172410: Replace the current multisource analytics-store setup.Mar 4 2019, 6:46 AM
Stashbot subscribed.

Mentioned in SAL (#wikimedia-operations) [2019-03-04T06:46:20Z] <marostegui> Stop MySQL on dbstore1002 for decommission T210478 T172410 T216491 T215589

Stashbot mentioned this in T215589: Migrate users to dbstore100[3-5].Mar 4 2019, 6:46 AM
gerritbot added a comment.Mar 4 2019, 6:48 AM

Change 494164 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Delete analytics-store CNAME due to dbstore1002 decom

https://gerrit.wikimedia.org/r/494164

gerritbot added a comment.Mar 4 2019, 6:49 AM

Change 494164 merged by Elukey:
[operations/dns@master] Delete analytics-store CNAME due to dbstore1002 decom

https://gerrit.wikimedia.org/r/494164

Marostegui added a comment.Mar 4 2019, 6:52 AM

MySQL has been stopped on dbstore1002 and won't be started again, as this host will be decommissioned

Marostegui changed the task status from Stalled to Open.Mar 4 2019, 6:53 AM
Marostegui updated the task description. (Show Details)
Stashbot added a comment.Mar 4 2019, 7:13 AM

Mentioned in SAL (#wikimedia-operations) [2019-03-04T07:13:05Z] <marostegui> Remove dbstore1002 from tendril and zarcillo - T216491

• jcrespo awarded a token.Mar 4 2019, 9:27 AM
elukey updated the task description. (Show Details)EditedMar 6 2019, 7:15 AM

No complaints or outages after the shutdown of dbstore1002, I think that we are good to keep going with the decom.

"@Marostegui this is control tower, you are clear to engage"

Marostegui added a comment.Mar 6 2019, 7:21 AM

\o/
Clear to decommission! :)

gerritbot added a comment.Mar 6 2019, 7:25 AM

Change 494649 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbstore1002: Set it to spare

https://gerrit.wikimedia.org/r/494649

gerritbot added a comment.Mar 6 2019, 7:32 AM

Change 494649 merged by Marostegui:
[operations/puppet@production] dbstore1002: Set it to spare

https://gerrit.wikimedia.org/r/494649

Stashbot added a comment.Mar 6 2019, 7:34 AM

Mentioned in SAL (#wikimedia-operations) [2019-03-06T07:34:07Z] <marostegui> Remove dbstore1002 from tendril and zarcillo T216491

Marostegui assigned this task to RobH.Mar 6 2019, 7:37 AM
Marostegui updated the task description. (Show Details)

Ready for @RobH to do the next steps.
@RobH can you give this some priority for the steps that include the power down of this host? It is a trusty host and it will soon stop receiving updates, so probably good to have it down (the onsite steps are not that urgent) in order to avoid having a host with no security updates ON.
Thanks!

Marostegui closed subtask T210478: Migrate dbstore1002 to a multi instance setup on dbstore100[3-5] as Resolved.Mar 6 2019, 8:38 AM
RobH edited projects, added decommission-hardware; removed Patch-For-Review.Mar 6 2019, 4:08 PM
RobH moved this task from Backlog to Ready for Decommission on the decommission-hardware board.
ops-monitoring-bot subscribed.Mar 6 2019, 5:40 PM

wmf-decommission-host was executed by robh for dbstore1002.eqiad.wmnet and performed the following actions:

  • Revoked Puppet certificate
  • Removed from PuppetDB
  • Downtimed host on Icinga
  • Downtimed mgmt interface on Icinga
  • Removed from DebMonitor
RobH updated the task description. (Show Details)Mar 6 2019, 5:41 PM
gerritbot added a comment.Mar 6 2019, 5:43 PM

Change 494794 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom dbstore1002

https://gerrit.wikimedia.org/r/494794

gerritbot added a project: Patch-For-Review.Mar 6 2019, 5:43 PM

Change 494795 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom dbstore1002

https://gerrit.wikimedia.org/r/494795

gerritbot added a comment.Mar 6 2019, 5:45 PM

Change 494795 merged by RobH:
[operations/dns@master] decom dbstore1002

https://gerrit.wikimedia.org/r/494795

gerritbot added a comment.Mar 6 2019, 5:45 PM

Change 494794 merged by RobH:
[operations/puppet@production] decom dbstore1002

https://gerrit.wikimedia.org/r/494794

RobH reassigned this task from RobH to • Cmjohnson.Mar 6 2019, 5:46 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH moved this task from Ready for Decommission to pending onsite steps (eqiad) on the decommission-hardware board.
gerritbot added a comment.Mar 7 2019, 9:10 AM

Change 494880 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Remove mariadb::dbstore

https://gerrit.wikimedia.org/r/494880

gerritbot added a project: Patch-For-Review.Mar 7 2019, 9:10 AM
gerritbot added a comment.Mar 7 2019, 9:43 AM

Change 494880 merged by Marostegui:
[operations/puppet@production] mariadb: Remove mariadb::dbstore

https://gerrit.wikimedia.org/r/494880

gerritbot added a comment.Mar 7 2019, 9:51 AM

Change 494886 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] aliases.yaml.erb: Remove dbstore

https://gerrit.wikimedia.org/r/494886

gerritbot added a comment.Mar 7 2019, 9:57 AM

Change 494886 merged by Marostegui:
[operations/puppet@production] aliases.yaml.erb: Remove dbstore

https://gerrit.wikimedia.org/r/494886

• Tbayer mentioned this in T217926: Import of wmfdata fails while trying to access analytics-store .Mar 8 2019, 10:24 PM
Jclark-ctr claimed this task.Oct 15 2019, 8:57 PM
Jclark-ctr added a subscriber: • Cmjohnson.
Jclark-ctr updated the task description. (Show Details)Nov 1 2019, 10:37 PM
Papaul subscribed.Nov 8 2019, 5:40 PM
papaul@asw2-d-eqiad# show | compare 
[edit interfaces]
-   ge-1/0/7 {
-       description dbstore1002;
-       enable;
-   }
Papaul updated the task description. (Show Details)Nov 8 2019, 5:41 PM
gerritbot added a comment.Nov 8 2019, 6:45 PM

Change 549912 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Remove mgmt DNS for analytic1003, dbstore1002 abd ms-be1027

https://gerrit.wikimedia.org/r/549912

gerritbot added a comment.Nov 8 2019, 6:53 PM

Change 549912 merged by Papaul:
[operations/dns@master] DNS: Remove mgmt DNS for analytic1003, dbstore1002 abd ms-be1027

https://gerrit.wikimedia.org/r/549912

Papaul closed this task as Resolved.Nov 8 2019, 7:02 PM
Papaul updated the task description. (Show Details)

Complete

Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:44 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits