Page MenuHomePhabricator

Decommission old coredb machines (<=db1050)
Closed, ResolvedPublic

Description

Old DB machines are out of warranty and should not be used for production services, they need to be effectively put out of service. Also, some needs to be unracked in order to rack the new ones that replace them.

However, before decommission of those old dbs, some of them were old masters, which means they probably hold the most accurate db schema- so we must check there are no data differences with the current masters and any other slaves before deleting its data.

Also, fix the issue with 'mysql' user account not being a system account (uid>999, or whatever is the configuration). It mostly affects these older machines.

These machines are no longer on production, please proceed now

Related Objects

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusAssignedTask
ResolvedNone
Resolvedjcrespo
ResolvedMarostegui
Resolvedjcrespo
ResolvedNone
ResolvedMarostegui
ResolvedCmjohnson
Resolvedelukey
ResolvedCmjohnson
Resolvedjcrespo
ResolvedMarostegui
ResolvedNone
Resolvedjcrespo
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedMarostegui
ResolvedMarostegui
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
Resolvedjcrespo
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
Resolvedjcrespo
Resolvedjcrespo
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedRobH
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Cmjohnson closed subtask T174902: Decommission db1037 as Resolved.
Cmjohnson closed subtask T177911: Decommission db1038 as Resolved.
Cmjohnson closed subtask T173915: Decommission db1041 as Resolved.

Change 393023 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Move db1053 to s2

https://gerrit.wikimedia.org/r/393023

Change 393023 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Move db1053 to s2

https://gerrit.wikimedia.org/r/393023

Change 393024 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1053 to s2

https://gerrit.wikimedia.org/r/393024

Mentioned in SAL (#wikimedia-operations) [2017-11-23T07:16:13Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Move db1053 to s2 to replace db1021 as vslow, dump slave - T134476 (duration: 00m 45s)

Change 393024 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1053 to s2

https://gerrit.wikimedia.org/r/393024

Change 393025 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1021

https://gerrit.wikimedia.org/r/393025

Change 393025 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1021

https://gerrit.wikimedia.org/r/393025

Mentioned in SAL (#wikimedia-operations) [2017-11-23T07:28:51Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1021 - T134476 (duration: 00m 45s)

Change 393069 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Pool db1053 as vslow,dump in s2

https://gerrit.wikimedia.org/r/393069

Change 393069 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Pool db1053 as vslow,dump in s2

https://gerrit.wikimedia.org/r/393069

Mentioned in SAL (#wikimedia-operations) [2017-11-23T14:09:14Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Pool db1053 in s2 as vslow to replace db1021 - T134476 (duration: 00m 45s)

jcrespo edited projects, added Goal; removed Patch-For-Review.Dec 21 2017, 10:19 AM

Change 399601 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool db1067, remove references to db1055, 56, 39

https://gerrit.wikimedia.org/r/399601

Change 399601 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Remove references to db1055 & db1056

https://gerrit.wikimedia.org/r/399601

Change 399792 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Decommissioning proposal

https://gerrit.wikimedia.org/r/399792

Marostegui mentioned this in Unknown Object (Task).Mar 6 2018, 7:08 AM

Change 420978 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Cleanup hosts that are no longer part of production

https://gerrit.wikimedia.org/r/420978

Change 420978 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Cleanup hosts that are no longer part of production

https://gerrit.wikimedia.org/r/420978

Marostegui added a subscriber: mark.

All the hosts <=db1050 have now been retired from service and are just pending to be decommissioned by DC Ops - they have their own individual decommissioning tasks, so I think we can consider this epic task resolved! //cc @mark

jcrespo updated the task description. (Show Details)
Marostegui moved this task from Meta/Epic to Done on the DBA board.Mar 22 2018, 10:59 AM
Marostegui added a subscriber: RobH.

I talked to @mark and we'll leave this ticket open until they have been fully decommissioned by @RobH and @Cmjohnson

Change 399792 abandoned by Jcrespo:
mariadb: Decommissioning proposal

Reason:
As agreed

https://gerrit.wikimedia.org/r/399792

@Cmjohnson and @RobH Thanks for all the hard work on eqiad!- once all decommission steps happen (we can and should wait for it to finish, that is more important for us) we should start next thinking about racking and setting up the new 8 eqiad db servers.

Marostegui closed this task as Resolved.Jun 26 2018, 5:36 PM

All the hosts are now totally decommissioned. So this is all done!
Thanks everyone for getting all these hosts decommissioned!!

Change 454291 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mysql user: Remove exception for mysql user being removed

https://gerrit.wikimedia.org/r/454291

Change 454291 merged by Jcrespo:
[operations/puppet@production] mysql user: Remove exception for mysql user being removed

https://gerrit.wikimedia.org/r/454291