Page MenuHomePhabricator
Create Task
Maniphest T134476

Decommission old coredb machines (<=db1050)
Closed, ResolvedPublic
Actions

Description

Old DB machines are out of warranty and should not be used for production services, they need to be effectively put out of service. Also, some needs to be unracked in order to rack the new ones that replace them.

However, before decommission of those old dbs, some of them were old masters, which means they probably hold the most accurate db schema- so we must check there are no data differences with the current masters and any other slaves before deleting its data.

Also, fix the issue with 'mysql' user account not being a system account (uid>999, or whatever is the configuration). It mostly affects these older machines.

These machines are no longer on production, please proceed now

Details

SubjectRepoBranchLines +/-
mysql user: Remove exception for mysql user being removedoperations/puppetproduction+0 -1
mariadb: Decommissioning proposaloperations/mediawiki-configmaster+22 -32
mariadb: Cleanup hosts that are no longer part of productionoperations/mediawiki-configmaster+0 -46
mariadb: Remove references to db1055 & db1056operations/mediawiki-configmaster+0 -16
db-eqiad.php: Pool db1053 as vslow,dump in s2operations/mediawiki-configmaster+7 -7
db-eqiad.php: Depool db1021operations/mediawiki-configmaster+6 -6
mariadb: Move db1053 to s2operations/puppetproduction+4 -4
db-eqiad.php: Move db1053 to s2operations/mediawiki-configmaster+1 -6
Depool db1023, 24, 33, 35, 39, 44, 52, 61, 64, 68, 63, 67, 72, 73operations/mediawiki-configmaster+15 -15
Prepare old servers for decom by sending all queries to new serversoperations/mediawiki-configmaster+36 -36
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedJoeT154658 Prepare and improve the datacenter switchover procedure
 ResolvedNoneT155099 Database maintenance scheduled while eqiad datacenter is non primary (after the DC switchover)
Restricted Task
 OpenNoneT157702 Followup for TLS MariaDB server roll-out
 ResolvedNoneT134476 Decommission old coredb machines (<=db1050)
 ResolvedjcrespoT138810 Set databases as read-only or switchover to secondary datacenter
 ResolvedMarosteguiT146265 db1019: Decommission
 ResolvedjcrespoT139280 db1034 decommission
 ResolvedNoneT129395 Decommission db1010
Restricted Task
 ResolvedMarosteguiT148078 Decommission db1015, db1035, db1044 and db1038
 ResolvedMarosteguiT164488 Run pt-table-checksum on s3
 Resolved• CmjohnsonT177911 Decommission db1038
 Resolved• CmjohnsonT149793 Decommission db1042
 ResolvedelukeyT156844 Decommission old dbstore hosts (db1046, db1047)
Unknown Object (Task)
 ResolvedelukeyT177405 rack and setup db1107 and db1108
 Resolved• CmjohnsonT105768 Remove db1002-db1007 from production
 ResolvedjcrespoT160509 run pt-tablechecksum on s6
 ResolvedMarosteguiT161294 run pt-tablechecksum on s5/s8
 ResolvedNoneT162699 Decomissions old s2 eqiad hosts (db1018, db1021, db1024, db1036)
 ResolvedjcrespoT162133 Replace some masters in eqiad while it is not active
ResolvedMarosteguiT163109 Reclone db1063 to become a slave in s5
 ResolvedjcrespoT163110 Reclone db1068 to become master in s4
 ResolvedMarosteguiT163665 Reclone db1062 from db1041 (s7 master)
 Resolved• CmjohnsonT163778 Decommission db1022 (Was: db1022 broke while changing topology on s6- evaluate if to fix or directly decommission)
 ResolvedjcrespoT154485 run pt-table-checksum on s2 (WAS: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038)
 Resolved• CmjohnsonT164702 Decommission db1024
 Resolved• CmjohnsonT176215 decommission db1018
 Resolved• CmjohnsonT176311 decommission db1036
 Resolved• CmjohnsonT147309 Decommission db1019
 ResolvedMarosteguiT163190 Checksum data on s7
 ResolvedMarosteguiT165743 frwiktionary on s7 still needs fixing on the revision table
 ResolvedMarosteguiT162807 Run pt-table-checksum on s1 (enwiki)
 DeclinedMarosteguiT186503 Rebuild user_newtalk on db1052
 Resolved• CmjohnsonT164057 Decommission db1040
 Resolved• CmjohnsonT166486 Decommission db1023
 Resolved• CmjohnsonT173570 Decommission db1015
 Resolved• CmjohnsonT173915 Decommission db1041
 Resolved• CmjohnsonT174076 Decommission db1033 and db1028
 Resolved• CmjohnsonT175679 Decommission db1048 (was Move m3 slave to db1059)
 ResolvedjcrespoT162593 Run pt-table-checksum on s4 (commonswiki)
 Resolved• CmjohnsonT175264 Decommission db1049
 Resolved• CmjohnsonT176931 Decommission db1035
 Resolved• CmjohnsonT174902 Decommission db1037
 Resolved• CmjohnsonT178162 Decommission db1050
 Resolved• CmjohnsonT181378 Decommission db1021
 Resolved• CmjohnsonT181696 Decommission db1044
 Resolved• CmjohnsonT182556 Decommission db1034
 ResolvedjcrespoT183249 Reimage and upgrade to stretch all dbproxies
 ResolvedjcrespoT183469 Setup newer machines and replace all old misc (m*) and x1 eqiad machines
 Resolved• CmjohnsonT184054 Decommission db1029 and db1031
 Resolved• CmjohnsonT187542 Decommission db1043
 Resolved• CmjohnsonT184703 Decommission db1011
 ResolvedjcrespoT184704 Setup tendril database monitoring on 2 new hosts, one on eqiad and one on codfw
Unknown Object (Task)
 ResolvedMarosteguiT186123 rack/setup/install db2093 (WAS: rack/setup/install tendril2001)
 ResolvedRobHT186172 switch port configuration for db2093
 ResolvedAndrewT189005 Failover m5 master from db1009 to db1073
 Resolved• CmjohnsonT189216 Decommission db1009
 ResolvedjcrespoT189655 Switchover m1 master from db1016 to db1063
 Resolved• CmjohnsonT190179 Decommission db1016
 Resolved• CmjohnsonT189773 Decommission db1020
 ResolvedjcrespoT189656 Switchover m2 master from db1020 to db1051
 Resolved• CmjohnsonT190262 Decommission db1001
 ResolvedRobHT184262 Decommission db1039
 Resolved• CmjohnsonT184397 Decommission db1030

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
jcrespo closed subtask T162593: Run pt-table-checksum on s4 (commonswiki) as Resolved.Sep 12 2017, 11:50 AM
jcrespo added a subtask: T175679: Decommission db1048 (was Move m3 slave to db1059).Sep 12 2017, 12:05 PM
jcrespo added a subtask: T175264: Decommission db1049.Sep 14 2017, 10:50 AM
jcrespo created subtask T176215: decommission db1018.Sep 19 2017, 2:14 PM
jcrespo mentioned this in T176243: Decommission database hosts <= db2031 (tracking).Sep 19 2017, 6:27 PM
jcrespo removed a subtask: T176215: decommission db1018.Sep 20 2017, 11:24 AM
Marostegui mentioned this in T131146: Analyze memory usage on db1011.Oct 6 2017, 6:29 AM
Marostegui added a subtask: T177911: Decommission db1038.Oct 11 2017, 8:17 AM
Marostegui added a subtask: T176931: Decommission db1035.
Marostegui added a subtask: T174902: Decommission db1037.Oct 11 2017, 8:19 AM
• Cmjohnson closed subtask T164702: Decommission db1024 as Resolved.Oct 12 2017, 4:48 PM
• Cmjohnson closed subtask T166486: Decommission db1023 as Resolved.Oct 12 2017, 5:06 PM
Marostegui added a subtask: T178162: Decommission db1050.Oct 13 2017, 12:46 PM
• Cmjohnson closed subtask T163778: Decommission db1022 (Was: db1022 broke while changing topology on s6- evaluate if to fix or directly decommission) as Resolved.Oct 16 2017, 5:07 PM
• Cmjohnson closed subtask T174076: Decommission db1033 and db1028 as Resolved.Nov 14 2017, 4:21 PM
• Cmjohnson closed subtask T176931: Decommission db1035 as Resolved.Nov 14 2017, 4:23 PM
• Cmjohnson closed subtask T174902: Decommission db1037 as Resolved.
• Cmjohnson closed subtask T177911: Decommission db1038 as Resolved.
• Cmjohnson closed subtask T173915: Decommission db1041 as Resolved.
gerritbot added a comment.Nov 23 2017, 7:10 AM

Change 393023 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Move db1053 to s2

https://gerrit.wikimedia.org/r/393023

gerritbot added a comment.Nov 23 2017, 7:14 AM

Change 393023 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Move db1053 to s2

https://gerrit.wikimedia.org/r/393023

gerritbot added a comment.Nov 23 2017, 7:16 AM

Change 393024 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Move db1053 to s2

https://gerrit.wikimedia.org/r/393024

Stashbot subscribed.Nov 23 2017, 7:16 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-23T07:16:13Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Move db1053 to s2 to replace db1021 as vslow, dump slave - T134476 (duration: 00m 45s)

gerritbot added a comment.Nov 23 2017, 7:20 AM

Change 393024 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1053 to s2

https://gerrit.wikimedia.org/r/393024

gerritbot added a comment.Nov 23 2017, 7:23 AM

Change 393025 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1021

https://gerrit.wikimedia.org/r/393025

gerritbot added a comment.Nov 23 2017, 7:27 AM

Change 393025 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1021

https://gerrit.wikimedia.org/r/393025

Stashbot added a comment.Nov 23 2017, 7:28 AM

Mentioned in SAL (#wikimedia-operations) [2017-11-23T07:28:51Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1021 - T134476 (duration: 00m 45s)

gerritbot added a comment.Nov 23 2017, 2:01 PM

Change 393069 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Pool db1053 as vslow,dump in s2

https://gerrit.wikimedia.org/r/393069

gerritbot added a comment.Nov 23 2017, 2:04 PM

Change 393069 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Pool db1053 as vslow,dump in s2

https://gerrit.wikimedia.org/r/393069

Stashbot added a comment.Nov 23 2017, 2:09 PM

Mentioned in SAL (#wikimedia-operations) [2017-11-23T14:09:14Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Pool db1053 in s2 as vslow to replace db1021 - T134476 (duration: 00m 45s)

Marostegui created subtask T181378: Decommission db1021.Nov 27 2017, 6:45 AM
Marostegui created subtask T181696: Decommission db1044.Nov 30 2017, 7:45 AM
Marostegui closed subtask T148078: Decommission db1015, db1035, db1044 and db1038 as Resolved.
Marostegui closed subtask T162699: Decomissions old s2 eqiad hosts (db1018, db1021, db1024, db1036) as Resolved.Dec 5 2017, 6:29 AM
Marostegui closed subtask T156844: Decommission old dbstore hosts (db1046, db1047) as Resolved.Dec 11 2017, 1:40 PM
jcrespo edited projects, added Goal; removed Patch-For-Review.Dec 21 2017, 10:19 AM
gerritbot added a comment.Dec 21 2017, 10:33 AM

Change 399601 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool db1067, remove references to db1055, 56, 39

https://gerrit.wikimedia.org/r/399601

gerritbot added a project: Patch-For-Review.Dec 21 2017, 10:33 AM
gerritbot added a comment.Dec 21 2017, 10:49 AM

Change 399601 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Remove references to db1055 & db1056

https://gerrit.wikimedia.org/r/399601

gerritbot added a comment.Dec 22 2017, 9:40 AM

Change 399792 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Decommissioning proposal

https://gerrit.wikimedia.org/r/399792

Marostegui closed subtask T161294: run pt-tablechecksum on s5/s8 as Resolved.Dec 27 2017, 9:23 AM
• Cmjohnson closed subtask T178162: Decommission db1050 as Resolved.Jan 3 2018, 7:15 PM
Marostegui closed subtask T163190: Checksum data on s7 as Resolved.Jan 5 2018, 3:36 PM
Marostegui added a subtask: T184397: Decommission db1030.Jan 8 2018, 8:05 AM
• Cmjohnson closed subtask T181696: Decommission db1044 as Resolved.Jan 10 2018, 3:47 PM
• Cmjohnson closed subtask T175679: Decommission db1048 (was Move m3 slave to db1059) as Resolved.
• Cmjohnson closed subtask T173570: Decommission db1015 as Resolved.
• Cmjohnson closed subtask T175264: Decommission db1049 as Resolved.
• Cmjohnson closed subtask T181378: Decommission db1021 as Resolved.
jcrespo changed the status of subtask T183249: Reimage and upgrade to stretch all dbproxies from Open to Stalled.Jan 10 2018, 6:20 PM
jcrespo added a subtask: T184703: Decommission db1011.Jan 11 2018, 11:31 AM
jcrespo reopened subtask T154485: run pt-table-checksum on s2 (WAS: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038) as Open.Jan 17 2018, 12:16 PM
jcrespo closed subtask T154485: run pt-table-checksum on s2 (WAS: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038) as Resolved.Jan 17 2018, 6:20 PM
jcrespo reopened subtask T160509: run pt-tablechecksum on s6 as Open.Jan 18 2018, 8:58 AM
jcrespo closed subtask T160509: run pt-tablechecksum on s6 as Resolved.Jan 18 2018, 12:20 PM
jcrespo reopened subtask T163190: Checksum data on s7 as Open.Jan 18 2018, 4:28 PM
jcrespo closed subtask T163190: Checksum data on s7 as Resolved.Jan 19 2018, 8:25 AM
jcrespo removed a subtask: T183470: Setup newer machines and replace all old misc (m*) and x1 codfw machines.Feb 20 2018, 7:05 PM
jcrespo removed a subtask: T184703: Decommission db1011.Feb 23 2018, 5:45 PM
Marostegui mentioned this in Unknown Object (Task).Mar 6 2018, 7:08 AM
Marostegui added a subtask: T189216: Decommission db1009.Mar 8 2018, 5:41 PM
Marostegui closed subtask T162807: Run pt-table-checksum on s1 (enwiki) as Resolved.Mar 13 2018, 9:00 AM
jcrespo changed the status of subtask T183249: Reimage and upgrade to stretch all dbproxies from Stalled to Open.Mar 14 2018, 3:05 PM
jcrespo removed a subtask: T189773: Decommission db1020.Mar 15 2018, 12:33 PM
jcrespo added a subtask: T189773: Decommission db1020.Mar 15 2018, 12:38 PM
jcrespo closed subtask T183249: Reimage and upgrade to stretch all dbproxies as Resolved.Mar 16 2018, 6:20 PM
Marostegui added a subtask: T190179: Decommission db1016.Mar 20 2018, 4:23 PM
gerritbot added a comment.Mar 21 2018, 10:22 AM

Change 420978 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Cleanup hosts that are no longer part of production

https://gerrit.wikimedia.org/r/420978

gerritbot added a comment.Mar 21 2018, 11:20 AM

Change 420978 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Cleanup hosts that are no longer part of production

https://gerrit.wikimedia.org/r/420978

Marostegui closed subtask T183469: Setup newer machines and replace all old misc (m*) and x1 eqiad machines as Resolved.Mar 22 2018, 6:41 AM
Marostegui added a subscriber: mark.

All the hosts <=db1050 have now been retired from service and are just pending to be decommissioned by DC Ops - they have their own individual decommissioning tasks, so I think we can consider this epic task resolved! //cc @mark

jcrespo edited projects, added hardware-requests; removed Patch-For-Review.Mar 22 2018, 7:44 AM
jcrespo updated the task description. (Show Details)
Marostegui moved this task from Meta/Epic to Done on the DBA board.Mar 22 2018, 10:59 AM
Marostegui added a subscriber: RobH.

I talked to @mark and we'll leave this ticket open until they have been fully decommissioned by @RobH and @Cmjohnson

• Cmjohnson closed subtask T189216: Decommission db1009 as Resolved.Mar 23 2018, 4:38 PM
Marostegui added a subtask: T187542: Decommission db1043.Mar 23 2018, 4:41 PM
Marostegui added a subtask: T184703: Decommission db1011.
Marostegui added a subtask: T184054: Decommission db1029 and db1031.
gerritbot added a comment.Apr 2 2018, 7:32 AM

Change 399792 abandoned by Jcrespo:
mariadb: Decommissioning proposal

Reason:
As agreed

https://gerrit.wikimedia.org/r/399792

• Cmjohnson closed subtask T190262: Decommission db1001 as Resolved.Apr 3 2018, 6:33 PM
• Cmjohnson closed subtask T190179: Decommission db1016 as Resolved.Apr 3 2018, 6:36 PM
• Cmjohnson closed subtask T184703: Decommission db1011 as Resolved.Apr 3 2018, 6:40 PM
• Cmjohnson closed subtask T184397: Decommission db1030 as Resolved.
jcrespo added a comment.Apr 4 2018, 10:54 AM

@Cmjohnson and @RobH Thanks for all the hard work on eqiad!- once all decommission steps happen (we can and should wait for it to finish, that is more important for us) we should start next thinking about racking and setting up the new 8 eqiad db servers.

• Cmjohnson closed subtask T189773: Decommission db1020 as Resolved.Apr 4 2018, 7:04 PM
RobH moved this task from Backlog to Reclaim (Spares/Decommission) on the hardware-requests board.Apr 11 2018, 5:49 PM
RobH edited projects, added decommission-hardware; removed hardware-requests.May 3 2018, 10:42 PM
• Cmjohnson closed subtask T184262: Decommission db1039 as Resolved.May 7 2018, 5:13 PM
• Cmjohnson closed subtask T182556: Decommission db1034 as Resolved.
• Cmjohnson closed subtask T184054: Decommission db1029 and db1031 as Resolved.Jun 26 2018, 4:44 PM
Marostegui closed this task as Resolved.Jun 26 2018, 5:36 PM

All the hosts are now totally decommissioned. So this is all done!
Thanks everyone for getting all these hosts decommissioned!!

gerritbot added a comment.Aug 21 2018, 2:36 PM

Change 454291 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mysql user: Remove exception for mysql user being removed

https://gerrit.wikimedia.org/r/454291

gerritbot added a project: Patch-For-Review.Aug 21 2018, 2:36 PM
gerritbot added a comment.Sep 18 2018, 7:36 AM

Change 454291 merged by Jcrespo:
[operations/puppet@production] mysql user: Remove exception for mysql user being removed

https://gerrit.wikimedia.org/r/454291

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL