Page MenuHomePhabricator
Create Task
Maniphest T394371

Migrate to MariaDB 10.11
Closed, ResolvedPublic
Actions

Description

We've been running 10.11 in msX, pcX and lost of es* (RO), with no issues.
Apart from that we've had hosts running on sX sections running 10.11 and it has all been fine. We are ready to slowly keep migrating sections to MariaDB 10.11

Ideally, the new debian OS version will be installed with MariaDB 10.11 as we usually do in between versions.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
mariadb: Upgrade db2202 (test-s1) to mariadb 10.11operations/puppetproduction+1 -0
db1215: Migrate to MariaDB 10.11operations/puppetproduction+1 -0
db2185: Migrate to MariaDB 10.11operations/puppetproduction+1 -0
db1155: Migrate to MariaDB 10.11operations/puppetproduction+1 -0
db1217: Migrate to MariaDB 10.11operations/puppetproduction+1 -0
mariadb: Test 10.11 backups on db1239 (standby backup source)operations/puppetproduction+2 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedMarosteguiT397425 Build 10.6.22 and 10.11.13 with mdev36934 patch
 ResolvedMarosteguiT394371 Migrate to MariaDB 10.11
 ResolvedMarosteguiT387953 Migrate pc1 to MariaDB 10.11
 ResolvedfnegriT394372 Migrate clouddb* hosts to MariaDB 10.11
 ResolvedBTullisT394930 an-redacteddb1001: upgrade MariaDB to 10.11
 ResolvedMarosteguiT394373 Migrate dbstore* hosts to MariaDB 10.11
 ResolvedMarosteguiT391921 Migrate read only external store to MariaDB 10.11
 ResolvedMarosteguiT394469 Migrate es6 to MariaDB 10.11
 ResolvedMarosteguiT395420 Switchover es6 master (es2037 -> es2035)
 ResolvedJclark-ctrT395424 es1038 not booting up after reboot
 ResolvedMarosteguiT395867 Switchover es6 master (es1037 -> es1038)
 ResolvedjcrespoT394487 Migrate backup sources to MariaDB 10.11
 ResolvedMarosteguiT395647 Migrate es7 to MariaDB 10.11
 ResolvedMarosteguiT395785 Switchover es7 master (es2038 -> es2039)
 ResolvedMarosteguiT395982 Switchover es7 master (es1035 -> es1039)
 ResolvedMarosteguiT395989 Migrate s6 to MariaDB 10.11
 ResolvedMarosteguiT396509 Switchover s6 master (db2229 -> db2214)
 ResolvedMarosteguiT397198 Switchover s6 master (db1173 -> db1201)
 ResolvedMarosteguiT396549 Migrate s2 to MariaDB 10.11
 ResolvedMarosteguiT396976 Switchover s2 master (db2207 -> db2204)
 ResolvedMarosteguiT398906 Switchover s2 master (db1162 -> db1222)
 ResolvedMarosteguiT397161 Migrate misc multi-instance hosts to MariaDB 10.11
 ResolvedMarosteguiT397279 Migrate x1 to MariaDB 10.11
 ResolvedMarosteguiT397419 Switchover x1 master (db2215 -> db2196)
 ResolvedMarosteguiT397612 Switchover x1 master (db1237 -> db1220)
 ResolvedVRiley-WMFT398794 db1237 is not booting up
 ResolvedMarosteguiT397412 Migrate m5 to MariaDB 10.11
 ResolvedMarosteguiT397602 Migrate m2 to MariaDB 10.11
 ResolvedMarosteguiT397633 Switchover m2 master db1250 -> db1228
 ResolvedMarosteguiT398805 Migrate m3 to MariaDB 10.11
 ResolvedMarosteguiT398818 Switchover m3 (phabricator) master db1213 -> db1250
 ResolvedMarosteguiT398928 Migrate s5 to MariaDB 10.11
 ResolvedMarosteguiT399280 Switchover s5 master (db2192 -> db2213)
 ResolvedMarosteguiT399446 Switchover s5 master (db1230 -> db1210)
 ResolvedMarosteguiT399060 Migrate m1 to MariaDB 10.11
 ResolvedMarosteguiT399172 Switchover m1 master (db1207 -> db1213)
 ResolvedMarosteguiT399298 Migrate x3 to MariaDB 10.11
 ResolvedMarosteguiT399456 Switchover x3 master (db2241 -> db2162)
 ResolvedMarosteguiT399699 Switchover x3 master (db1255 -> db1258)
 ResolvedMarosteguiT399540 Upgrade masters to 10.6.22 and 10.11.13 .2 update
 DeclinedMarosteguiT400028 Switchover es7 master (es1035 -> es1039)
 DeclinedMarosteguiT400027 Switchover es6 master (es1038 -> es1037)
 ResolvedFCeratto-WMFT402276 Switchover s2 master (db2204 -> db2207)
 ResolvedMarosteguiT399430 Move db1207 to s5
 ResolvedMarosteguiT399548 Migrate s3 to MariaDB 10.11
 DeclinedMarosteguiT399954 Switchover s3 master (db1223 -> db1189)
 ResolvedMarosteguiT399930 Switchover s3 master (db2209 -> db2205)
 ResolvedMarosteguiT399955 Migrate s7 to MariaDB 10.11
 ResolvedMarosteguiT400591 Switchover s7 master (db2218 -> db2220)
 ResolvedFCeratto-WMFT406008 Migrate s8 to 10.11
 ResolvedFCeratto-WMFT406469 Cookbook for major MariaDB version upgrade
 ResolvedMarosteguiT409299 Switchover s8 master (db1193 -> db1209)
 ResolvedFCeratto-WMFT409818 Switchover s8 master (db2165 -> db2161)
 ResolvedMarosteguiT406541 Migrate s4 to 10.11
 ResolvedMarosteguiT407176 Switchover s4 master (db1244 -> db1160)
 ResolvedMarosteguiT407177 Switchover s4 master (db2240 -> db2179)
 ResolvedMarosteguiT407463 Migrate s1 to 10.11
 ResolvedMarosteguiT407975 Switchover s1 master (db1163 -> db1184)
 ResolvedMarosteguiT409255 Switchover s1 master (db2203 -> db2212)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
jcrespo updated the task description. (Show Details)Jun 12 2025, 9:30 AM
gerritbot added a comment.Jun 17 2025, 11:17 AM

Change #1160088 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1217: Migrate to MariaDB 10.11

https://gerrit.wikimedia.org/r/1160088

gerritbot added a project: Patch-For-Review.Jun 17 2025, 11:17 AM
gerritbot added a comment.Jun 17 2025, 11:22 AM

Change #1160088 merged by Marostegui:

[operations/puppet@production] db1217: Migrate to MariaDB 10.11

https://gerrit.wikimedia.org/r/1160088

Maintenance_bot removed a project: Patch-For-Review.Jun 17 2025, 11:31 AM
gerritbot added a comment.Jun 17 2025, 1:19 PM

Change #1160137 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1155: Migrate to MariaDB 10.11

https://gerrit.wikimedia.org/r/1160137

gerritbot added a project: Patch-For-Review.Jun 17 2025, 1:19 PM

Change #1160137 merged by Marostegui:

[operations/puppet@production] db1155: Migrate to MariaDB 10.11

https://gerrit.wikimedia.org/r/1160137

Maintenance_bot removed a project: Patch-For-Review.Jun 17 2025, 1:30 PM
Marostegui closed subtask T395989: Migrate s6 to MariaDB 10.11 as Resolved.Jun 18 2025, 5:30 AM
Marostegui updated the task description. (Show Details)
Marostegui closed subtask T397161: Migrate misc multi-instance hosts to MariaDB 10.11 as Resolved.Jun 18 2025, 5:57 AM
Marostegui updated the task description. (Show Details)Jun 18 2025, 6:58 AM
fnegri closed subtask T394372: Migrate clouddb* hosts to MariaDB 10.11 as Resolved.Jun 18 2025, 9:26 AM
Marostegui updated the task description. (Show Details)Jun 19 2025, 8:45 AM
Marostegui closed subtask T397412: Migrate m5 to MariaDB 10.11 as Resolved.Jun 23 2025, 7:15 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Jun 23 2025, 11:06 AM
Marostegui closed subtask T397602: Migrate m2 to MariaDB 10.11 as Resolved.Jul 7 2025, 9:31 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Jul 7 2025, 9:55 AM
Marostegui closed subtask T397279: Migrate x1 to MariaDB 10.11 as Resolved.Jul 8 2025, 4:22 AM
Marostegui updated the task description. (Show Details)
Marostegui closed subtask T396549: Migrate s2 to MariaDB 10.11 as Resolved.Jul 8 2025, 4:52 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Jul 8 2025, 9:58 AM
Marostegui closed subtask T398805: Migrate m3 to MariaDB 10.11 as Resolved.Jul 9 2025, 6:38 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Jul 9 2025, 6:42 AM
Marostegui updated the task description. (Show Details)Jul 11 2025, 11:44 AM
Marostegui closed subtask T399060: Migrate m1 to MariaDB 10.11 as Resolved.Jul 14 2025, 6:34 AM
Marostegui updated the task description. (Show Details)
Marostegui closed subtask T399430: Move db1207 to s5 as Resolved.Jul 14 2025, 10:47 AM
Marostegui closed subtask T398928: Migrate s5 to MariaDB 10.11 as Resolved.Jul 15 2025, 9:25 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Jul 15 2025, 9:28 AM
Marostegui closed subtask T399298: Migrate x3 to MariaDB 10.11 as Resolved.Jul 17 2025, 6:34 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)Jul 18 2025, 1:32 PM
Ladsgroup closed subtask T399548: Migrate s3 to MariaDB 10.11 as Resolved.Sep 8 2025, 11:56 PM
Ladsgroup updated the task description. (Show Details)Sep 10 2025, 10:06 AM
Ladsgroup closed subtask T399955: Migrate s7 to MariaDB 10.11 as Resolved.Sep 10 2025, 11:36 AM
Ladsgroup subscribed.Sep 10 2025, 11:38 AM

I will try to write a cookbook for this. It's a pain to do manually/manuelly.

Ladsgroup updated the task description. (Show Details)Sep 10 2025, 11:38 AM
gerritbot added a comment.Sep 10 2025, 12:45 PM

Change #1186975 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] db2185: Migrate to MariaDB 10.11

https://gerrit.wikimedia.org/r/1186975

gerritbot added a project: Patch-For-Review.Sep 10 2025, 12:45 PM
Stashbot added a comment.Sep 10 2025, 12:46 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-10T12:46:48Z] <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2185.codfw.wmnet with reason: Glow up (T394371)

gerritbot added a comment.Sep 10 2025, 12:47 PM

Change #1186975 merged by Ladsgroup:

[operations/puppet@production] db2185: Migrate to MariaDB 10.11

https://gerrit.wikimedia.org/r/1186975

gerritbot added a comment.Sep 10 2025, 12:53 PM

Change #1186983 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] db1215: Migrate to MariaDB 10.11

https://gerrit.wikimedia.org/r/1186983

Stashbot mentioned this in T399540: Upgrade masters to 10.6.22 and 10.11.13 .2 update.Sep 10 2025, 12:55 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-10T12:55:02Z] <ladsgroup@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1215.eqiad.wmnet with reason: Glow up (T399540 T394371)

gerritbot added a comment.Sep 10 2025, 12:58 PM

Change #1186983 merged by Ladsgroup:

[operations/puppet@production] db1215: Migrate to MariaDB 10.11

https://gerrit.wikimedia.org/r/1186983

Ladsgroup updated the task description. (Show Details)Sep 10 2025, 1:11 PM
Maintenance_bot removed a project: Patch-For-Review.Sep 10 2025, 1:32 PM
jcrespo updated the task description. (Show Details)Sep 12 2025, 11:24 AM
gerritbot added a comment.Sep 12 2025, 11:26 AM

Change #1187773 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Upgrade db2202 (test-s1) to mariadb 10.11

https://gerrit.wikimedia.org/r/1187773

gerritbot added a project: Patch-For-Review.Sep 12 2025, 11:26 AM
gerritbot added a comment.Sep 12 2025, 11:30 AM

Change #1187773 merged by Jcrespo:

[operations/puppet@production] mariadb: Upgrade db2202 (test-s1) to mariadb 10.11

https://gerrit.wikimedia.org/r/1187773

Maintenance_bot removed a project: Patch-For-Review.Sep 12 2025, 11:31 AM
jcrespo updated the task description. (Show Details)Sep 12 2025, 11:40 AM
Ladsgroup updated the task description. (Show Details)Sep 30 2025, 9:40 AM
Ladsgroup mentioned this in T405476: list=logevents slow or times out for users with many log entries but no recent activity.Oct 6 2025, 8:59 PM
Marostegui updated the task description. (Show Details)Oct 7 2025, 7:44 AM
Marostegui closed subtask T406541: Migrate s4 to 10.11 as Resolved.Oct 16 2025, 4:54 AM
Marostegui updated the task description. (Show Details)Oct 16 2025, 5:52 AM
Marostegui updated the task description. (Show Details)
Marostegui claimed this task.Oct 16 2025, 5:56 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)
Marostegui added a comment.Oct 16 2025, 5:58 AM

@jcrespo just to confirm backup1-codfw is pending but backup1-eqiad is done, right?

jcrespo updated the task description. (Show Details)Oct 16 2025, 8:12 AM
In T394371#11279960, @Marostegui wrote:

@jcrespo just to confirm backup1-codfw is pending but backup1-eqiad is done, right?

It's done, I just tracked it on a different task: T376916

Marostegui added a comment.Oct 16 2025, 8:14 AM
In T394371#11280206, @jcrespo wrote:
In T394371#11279960, @Marostegui wrote:

@jcrespo just to confirm backup1-codfw is pending but backup1-eqiad is done, right?

It's done, I just tracked it on a different task: T376916

And also 10.11? Cause I don't see those hosts with the package on the hiera file, so just double checking

Marostegui added a comment.Oct 16 2025, 8:15 AM

Nevermind, I was looking at the wrong hosts. All clarified. Sorry for the confusion

jcrespo added a comment.Oct 16 2025, 8:16 AM

https://orchestrator.wikimedia.org/web/cluster/alias/backup1-codfw

jcrespo closed subtask T394487: Migrate backup sources to MariaDB 10.11 as Resolved.Nov 10 2025, 6:33 PM
Marostegui closed subtask T407463: Migrate s1 to 10.11 as Resolved.Nov 12 2025, 7:09 AM
Marostegui updated the task description. (Show Details)
FCeratto-WMF closed subtask T406008: Migrate s8 to 10.11 as Resolved.Nov 13 2025, 7:58 AM
Marostegui closed this task as Resolved.Nov 13 2025, 8:04 AM
Marostegui updated the task description. (Show Details)

Wohoo! This is completed! Thanks everyone

nshahquinn-wmf awarded a token.Nov 19 2025, 2:49 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits