Page MenuHomePhabricator
Create Task
Maniphest T189216

Decommission db1009
Closed, ResolvedPublic
Actions

Description

db1009 was the old m5 master, and it was failed over to db1073.
Wait a few days and then proceed to decommission it

Decommission Checklist

  • - all system services confirmed offline from production use - should be done by DBA team https://gerrit.wikimedia.org/r/#/c/420293/
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration - should be done by DBA team
  • - any service group puppet/heira/dsh config removed - should be done by DBA team
  • - remove site.pp (replace with role(spare::system) if system isn't shut down immediately during this process.) - should be done by DBA team: https://gerrit.wikimedia.org/r/#/c/420295/
  • - ping @chasemp to get the ACL for 10.64.0.13 cleaned up T189216#4060598 (and IRC)

START NON-INTERRUPPTABLE STEPS - please assign to @RobH for the non-interrupt steps

  • - disable puppet on host
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal) asw-a-eqiad:ge-2/0/8
  • - remove all remaining puppet references (include role::spare)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: add system to decommission tracking google sheet
  • - IF DECOM: mgmt dns entries removed.

Details

Related Gerrit Patches:
operations/dns : masterRemoving mgmt dns for db1009
operations/dns : masterdecom db1009 prod dns entries
operations/puppet : productiondecom db1009
operations/puppet : productionmariadb: Set db1009 to spare
operations/software : masterm5.hosts: Remove db1009
operations/mediawiki-config : masterdb-eqiad,db-codfw.php: Remove db1009 from config
operations/puppet : productionsite.pp: db1009 is not a master anymore

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 OpenNoneT120532 Use user-specific passwords for accessing EventLogging database
 OpenaaronT88445 MediaWiki active/active datacenter investigation and work (tracking)
 StalledNoneT134809 Apache <=> mariadb SSL/TLS for cross-datacenter writes
 OpenNoneT111653 Encrypt all the things
 ResolvedjcrespoT111654 Set up TLS for MariaDB replication
 OpenNoneT138562 Improve regular production database backups handling
 OpenNoneT151583 Use tls for dump backup generation
 OpenNoneT157702 Followup for TLS MariaDB server roll-out
 ResolvedNoneT134476 Decommission old coredb machines (<=db1050)
 ResolvedjcrespoT183469 Setup newer machines and replace all old misc (m*) and x1 eqiad machines
 ResolvedCmjohnsonT184054 Decommission db1029 and db1031
 ResolvedCmjohnsonT187542 Decommission db1043
 ResolvedCmjohnsonT184703 Decommission db1011
 ResolvedjcrespoT184704 Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw
Unknown Object (Task)
 ResolvedMarosteguiT186123 rack/setup/install db2093 (WAS: rack/setup/install tendril2001)
 ResolvedRobHT186172 switch port configuration for db2093
 ResolvedAndrewT189005 Failover m5 master from db1009 to db1073
 ResolvedCmjohnsonT189216 Decommission db1009
 ResolvedjcrespoT189655 Switchover m1 master from db1016 to db1063
 ResolvedCmjohnsonT190179 Decommission db1016
 ResolvedCmjohnsonT189773 Decommission db1020
 ResolvedjcrespoT189656 Switchover m2 master from db1020 to db1051
 ResolvedCmjohnsonT190262 Decommission db1001

Event Timeline

Marostegui triaged this task as Medium priority.Mar 8 2018, 4:11 PM
Marostegui created this task.
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui updated the task description. (Show Details)
gerritbot added a subscriber: gerritbot.Mar 8 2018, 4:14 PM

Change 417300 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] site.pp: db1009 is not a master anymore

https://gerrit.wikimedia.org/r/417300

gerritbot added a comment.Mar 8 2018, 4:17 PM

Change 417300 merged by Marostegui:
[operations/puppet@production] site.pp: db1009 is not a master anymore

https://gerrit.wikimedia.org/r/417300

Marostegui added a comment.EditedMar 8 2018, 5:02 PM

A backup of this host is placed at: es2001:/srv/backups/older/m5/db1009_binary_copy/db1009.tar.gz

Marostegui added a comment.Mar 12 2018, 6:29 AM

Due to: T189403 I have started mysql on db1009 and connected it to db1073 temporarly. It is only listening on 127.0.0.1 so nothing can actually connect to it, but I wanted to have it replicating from db1073 till db1073 gets the disk replaced and the raid rebuilt. Better be safe :-)

jcrespo added a comment.Mar 12 2018, 10:28 AM

I don't see it replicating on tendril (I didn't touch it there), is that expected?

Marostegui added a comment.Mar 14 2018, 3:37 PM

@jcrespo is it ok to proceed with this or you're still checking it?

jcrespo added a comment.Mar 14 2018, 3:38 PM

If you don't mind leaving it like that for some more time, so I can run pt-table-checksum on all misc sections?

Marostegui added a comment.Mar 14 2018, 3:39 PM

Sure - that's perfectly ok! :-)

Restricted Application added a project: Operations. · View Herald TranscriptMar 14 2018, 5:04 PM
jcrespo claimed this task.Mar 14 2018, 5:06 PM
jcrespo removed projects: hardware-requests, Patch-For-Review.

wait, robh, I will take this for now- not yet ready for decom.

RobH added a comment.Mar 14 2018, 5:07 PM

I wasn't taking it, was merely tagging in all decom requests with #hw-requests. I left it assigned to @Marostegui ;]

Marostegui added a comment.Mar 14 2018, 5:13 PM

Yeah I never use the other tags till we have it ready from the DBA side, to avoid all the noise for the DC Ops :)

jcrespo added a comment.Mar 14 2018, 5:14 PM

Oh, if they want the noise, they will get it here :-P

RobH added a comment.Mar 14 2018, 6:59 PM
In T189216#4050652, @Marostegui wrote:

Yeah I never use the other tags till we have it ready from the DBA side, to avoid all the noise for the DC Ops :)

no worries, I wasnt sure so I added. If it is something where you guys don't want us aware until its ready for us to work on, the project can be added later. (I just didn't want it forgotten ;)

Marostegui added a comment.EditedMar 17 2018, 7:17 AM

So, the checks finished and there were differences on testreduce_0715.results (173GB) table, between the following rows:

40590911 40650121

I have confirmed that this is not a pt-table-checksum false positive by dumping those rows with mysqldump and doing a diff.

There are 22 rows that differ (out of around 64M - so not too bad). It is probably not worth at all to fix those, so what I am going to do is:

  • Backup a binary copy of db1009
  • Backup a logical copy of testreduce_0715.results table
  • Backup a diff of the affected differences

After that I will go ahead and decommission this host.

Marostegui added a comment.Mar 19 2018, 9:44 AM

Backup a binary copy of db1009: es2001:/srv/backups/older/m5/db1009_binary_copy/db1009.tar.gz
Backup a logical copy of testreduce_0715.results table: es2001:/srv/backups/older/m5/db1009_results_table/db1009_testreduce_0715_results.sql
Backup a diff of the affected differences: es2001:/srv/backups/older/m5/diff_db1009_db1073.sql

Stashbot added a comment.Mar 19 2018, 9:45 AM

Mentioned in SAL (#wikimedia-operations) [2018-03-19T09:45:48Z] <marostegui> Stop MySQL on db1009 - T189216

gerritbot added a comment.Mar 19 2018, 9:48 AM

Change 420293 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1009 from config

https://gerrit.wikimedia.org/r/420293

gerritbot added a comment.Mar 19 2018, 9:51 AM

Change 420293 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Remove db1009 from config

https://gerrit.wikimedia.org/r/420293

Stashbot added a comment.Mar 19 2018, 9:54 AM

Mentioned in SAL (#wikimedia-operations) [2018-03-19T09:54:32Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1009 from config - T189216 (duration: 00m 58s)

Stashbot added a comment.Mar 19 2018, 9:55 AM

Mentioned in SAL (#wikimedia-operations) [2018-03-19T09:55:38Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1009 from config - T189216 (duration: 00m 57s)

Marostegui updated the task description. (Show Details)Mar 19 2018, 9:56 AM
gerritbot added a comment.Mar 19 2018, 10:21 AM

Change 420295 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Set db1009 to spare

https://gerrit.wikimedia.org/r/420295

gerritbot added a comment.Mar 19 2018, 10:21 AM

Change 420296 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] m5.hosts: Remove db1009

https://gerrit.wikimedia.org/r/420296

gerritbot added a comment.Mar 19 2018, 10:24 AM

Change 420296 merged by jenkins-bot:
[operations/software@master] m5.hosts: Remove db1009

https://gerrit.wikimedia.org/r/420296

Stashbot added a comment.Mar 19 2018, 10:25 AM

Mentioned in SAL (#wikimedia-operations) [2018-03-19T10:25:09Z] <marostegui> Remove db1009 from tendril - T189216

gerritbot added a comment.Mar 19 2018, 10:26 AM

Change 420295 merged by Marostegui:
[operations/puppet@production] mariadb: Set db1009 to spare

https://gerrit.wikimedia.org/r/420295

Marostegui updated the task description. (Show Details)Mar 19 2018, 10:27 AM

@chasemp can you please proceed and remove the ACL for db1009 now?

Marostegui reassigned this task from jcrespo to RobH.Mar 19 2018, 10:28 AM
Marostegui updated the task description. (Show Details)
Marostegui moved this task from In progress to Done on the DBA board.

This host is now ready for DC Ops decommissioning, so assigning it to @RobH

Marostegui added a subscriber: ssastry.Mar 19 2018, 3:07 PM
In T189216#4058198, @Marostegui wrote:

So, the checks finished and there were differences on testreduce_0715.results (173GB) table, between the following rows:

40590911 40650121

I have confirmed that this is not a pt-table-checksum false positive by dumping those rows with mysqldump and doing a diff.
There are 22 rows that differ (out of around 64M - so not too bad). It is probably not worth at all to fix those, so what I am going to do is:

  • Backup a binary copy of db1009
  • Backup a logical copy of testreduce_0715.results table
  • Backup a diff of the affected differences

After that I will go ahead and decommission this host.

I have talked to @ssastry and it has been confirmed that small drift isn't an issue.

RobH updated the task description. (Show Details)Mar 20 2018, 7:12 PM
gerritbot added a comment.Mar 20 2018, 7:15 PM

Change 420821 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] decom db1009

https://gerrit.wikimedia.org/r/420821

gerritbot added a comment.Mar 20 2018, 7:16 PM

Change 420821 merged by RobH:
[operations/puppet@production] decom db1009

https://gerrit.wikimedia.org/r/420821

gerritbot added a comment.Mar 20 2018, 7:19 PM

Change 420822 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] decom db1009 prod dns entries

https://gerrit.wikimedia.org/r/420822

gerritbot added a comment.Mar 20 2018, 7:20 PM

Change 420822 merged by RobH:
[operations/dns@master] decom db1009 prod dns entries

https://gerrit.wikimedia.org/r/420822

RobH reassigned this task from RobH to Cmjohnson.Mar 20 2018, 7:23 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
RobH moved this task from Backlog to Decommission on the ops-eqiad board.

ready for onsite disk wipe and completion of steps

Cmjohnson updated the task description. (Show Details)Mar 22 2018, 4:10 PM
gerritbot added a comment.Mar 23 2018, 4:36 PM

Change 421567 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Removing mgmt dns for db1009

https://gerrit.wikimedia.org/r/421567

gerritbot added a project: Patch-For-Review.Mar 23 2018, 4:37 PM

Change 421567 merged by Cmjohnson:
[operations/dns@master] Removing mgmt dns for db1009

https://gerrit.wikimedia.org/r/421567

Cmjohnson closed this task as Resolved.Mar 23 2018, 4:38 PM
Cmjohnson updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL