Migrate s8 to 10.11
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Ladsgroup
	Sep 30 2025, 9:35 AM

Description

See parent

Status:

+----------+----------------------+
| hostname | mariadb_version      |
+----------+----------------------+
| db2198   | 10.11.13-MariaDB-log |
| db2152   | 10.11.14-MariaDB-log |
| db2154   | 10.11.14-MariaDB-log |
| db2161   | 10.11.14-MariaDB-log |
| db2163   | 10.11.14-MariaDB-log |
| db2164   | 10.11.14-MariaDB-log |
| db2166   | 10.11.14-MariaDB-log |
| db2167   | 10.11.14-MariaDB-log |
| db2181   | 10.11.14-MariaDB-log |
| db2195   | 10.11.14-MariaDB-log |
| db2165   | 10.11.14-MariaDB-log |
+----------+----------------------+

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
db2165: Migration to MariaDB 10.11	operations/puppet	production	+2 -1
db2181: Migration to MariaDB 10.11	operations/puppet	production	+1 -0
db2167: Migration to MariaDB 10.11	operations/puppet	production	+1 -0
db2166: Migration to MariaDB 10.11	operations/puppet	production	+1 -0
db2164: Migration to MariaDB 10.11	operations/puppet	production	+1 -0
db2152: Migration to MariaDB 10.11	operations/puppet	production	+1 -0
db2154: Migration to MariaDB 10.11	operations/puppet	production	+1 -0
db2163: Migration to MariaDB 10.11	operations/puppet	production	+1 -0
db2152: Migration to MariaDB 10.11	operations/puppet	production	+1 -0
db2161: Migration to MariaDB 10.11	operations/puppet	production	+2 -1
migration1011.sh: Add to repo	operations/software	master	+38 -0
db1172: Upgrade to 10.11	operations/puppet	production	+1 -1

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	task_template*: Add some clarifications	toolforge-repos/switchmaster!15	marostegui	clarifications	main

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
Resolved	Marostegui	T394371 Migrate to MariaDB 10.11
Resolved	FCeratto-WMF	T406008 Migrate s8 to 10.11
Resolved	FCeratto-WMF	T406469 Cookbook for major MariaDB version upgrade
Resolved	Marostegui	T409299 Switchover s8 master (db1193 -> db1209)
Resolved	FCeratto-WMF	T409818 Switchover s8 master (db2165 -> db2161)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

FCeratto-WMF closed subtask T406469: Cookbook for major MariaDB version upgrade as Resolved.Oct 31 2025, 2:42 PM

I've migrated all s8 eqiad replicas

Marostegui added a subtask: T409299: Switchover s8 master (db1193 -> db1209).Nov 5 2025, 2:43 PM

Change #1202197 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] db2161: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202197

gerritbot added a project: Patch-For-Review.Nov 5 2025, 4:32 PM

Upgrading db2161.codfw.wmnet

Completed depool of db2161 - Upgrading db2161.codfw.wmnet - fceratto@cumin1003

Change #1202197 merged by Federico Ceratto:

[operations/puppet@production] db2161: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202197

Start pool of db2161 gradually with 4 steps - Migration of db2161.codfw.wmnet completed - fceratto@cumin1003

Maintenance_bot removed a project: Patch-For-Review.Nov 5 2025, 5:31 PM

Completed pool of db2161 gradually with 4 steps - Migration of db2161.codfw.wmnet completed - fceratto@cumin1003

Migration of db2161.codfw.wmnet completed

Marostegui closed subtask T409299: Switchover s8 master (db1193 -> db1209) as Resolved.Nov 6 2025, 7:24 AM

Marostegui moved this task from Ready to In progress on the DBA board.Nov 6 2025, 8:59 AM

Upgrading db2152.codfw.wmnet

Change #1202703 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] db2152: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202703

gerritbot added a project: Patch-For-Review.Nov 6 2025, 1:13 PM

Completed depool of db2152 - Upgrading db2152.codfw.wmnet - fceratto@cumin1003

Change #1202703 merged by Federico Ceratto:

[operations/puppet@production] db2152: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202703

Maintenance_bot removed a project: Patch-For-Review.Nov 6 2025, 1:30 PM

Start pool of db2152 gradually with 4 steps - Migration of db2152.codfw.wmnet completed - fceratto@cumin1003

Completed pool of db2152 gradually with 4 steps - Migration of db2152.codfw.wmnet completed - fceratto@cumin1003

Migration of db2152.codfw.wmnet completed

Change #1202724 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] db2163: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202724

gerritbot added a project: Patch-For-Review.Nov 6 2025, 2:37 PM

Upgrading db2163.codfw.wmnet

Completed depool of db2163 - Upgrading db2163.codfw.wmnet - fceratto@cumin1003

Change #1202724 merged by Federico Ceratto:

[operations/puppet@production] db2163: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202724

Start pool of db2163 gradually with 4 steps - Migration of db2163.codfw.wmnet completed - fceratto@cumin1003

Maintenance_bot removed a project: Patch-For-Review.Nov 6 2025, 3:30 PM

Completed pool of db2163 gradually with 4 steps - Migration of db2163.codfw.wmnet completed - fceratto@cumin1003

Migration of db2163.codfw.wmnet completed

Change #1202752 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] db2154: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202752

gerritbot added a project: Patch-For-Review.Nov 6 2025, 4:34 PM

Upgrading db2154.codfw.wmnet

Completed depool of db2154 - Upgrading db2154.codfw.wmnet - fceratto@cumin1003

Change #1202752 merged by Federico Ceratto:

[operations/puppet@production] db2154: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202752

Start pool of db2154 gradually with 4 steps - Migration of db2154.codfw.wmnet completed - fceratto@cumin1003

Maintenance_bot removed a project: Patch-For-Review.Nov 6 2025, 5:30 PM

Completed pool of db2154 gradually with 4 steps - Migration of db2154.codfw.wmnet completed - fceratto@cumin1003

Migration of db2154.codfw.wmnet completed

Change #1202773 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] db2152: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202773

Change #1202774 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] db2164: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202774

Change #1202775 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] db2166: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202775

Change #1202776 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] db2167: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202776

Change #1202777 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] db2181: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202777

Change #1202773 abandoned by Federico Ceratto:

[operations/puppet@production] db2152: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202773

Change #1202774 merged by Federico Ceratto:

[operations/puppet@production] db2164: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202774

Upgrading db2164.codfw.wmnet

Completed depool of db2164 - Upgrading db2164.codfw.wmnet - fceratto@cumin1003

Start pool of db2164 gradually with 4 steps - Migration of db2164.codfw.wmnet completed - fceratto@cumin1003

Completed pool of db2164 gradually with 4 steps - Migration of db2164.codfw.wmnet completed - fceratto@cumin1003

Migration of db2164.codfw.wmnet completed

FCeratto-WMF updated the task description. (Show Details)Nov 10 2025, 11:06 AM

Upgrading db2166.codfw.wmnet

Completed depool of db2166 - Upgrading db2166.codfw.wmnet - fceratto@cumin1003

Change #1202775 merged by Federico Ceratto:

[operations/puppet@production] db2166: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202775

Start pool of db2166 gradually with 4 steps - Migration of db2166.codfw.wmnet completed - fceratto@cumin1003

Completed pool of db2166 gradually with 4 steps - Migration of db2166.codfw.wmnet completed - fceratto@cumin1003

Migration of db2166.codfw.wmnet completed

FCeratto-WMF updated the task description. (Show Details)Nov 10 2025, 1:51 PM

Upgrading db2167.codfw.wmnet

Completed depool of db2167 - Upgrading db2167.codfw.wmnet - fceratto@cumin1003

Change #1202776 merged by Federico Ceratto:

[operations/puppet@production] db2167: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202776

FCeratto-WMF updated the task description. (Show Details)Nov 10 2025, 2:15 PM

Start pool of db2167 gradually with 4 steps - Migration of db2167.codfw.wmnet completed - fceratto@cumin1003

Completed pool of db2167 gradually with 4 steps - Migration of db2167.codfw.wmnet completed - fceratto@cumin1003

Migration of db2167.codfw.wmnet completed

Upgrading db2181.codfw.wmnet

Completed depool of db2181 - Upgrading db2181.codfw.wmnet - fceratto@cumin1003

Change #1202777 merged by Federico Ceratto:

[operations/puppet@production] db2181: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1202777

Start pool of db2181 gradually with 4 steps - Migration of db2181.codfw.wmnet completed - fceratto@cumin1003

Maintenance_bot removed a project: Patch-For-Review.Nov 10 2025, 5:31 PM

Completed pool of db2181 gradually with 4 steps - Migration of db2181.codfw.wmnet completed - fceratto@cumin1003

Migration of db2181.codfw.wmnet completed

FCeratto-WMF updated the task description. (Show Details)Nov 11 2025, 10:27 AM

In T406008#11233130, @Ladsgroup wrote:

The process is basically:

Depool and downtime

Create the puppet patch

Disable puppet

stop slave;

stop mariadb service

sudo apt remove wmf-mariadb106

merge puppet patch and run puppet-merge on puppetserver

enable puppet

run-puppet-agent

start mariadb service

mysql_upgrade (this is really important)

start slave;

once caught up, repool

As part of a proper major version upgrade, I would add a cleanup restart after mysql_upgrade. It makes sure logs come up clean and it makes sure we don't have any logging issue like we had in the past, new tables get reloaded satisfactorily.

Master flip in codfw: https://phabricator.wikimedia.org/T409818

FCeratto-WMF changed the status of subtask T409818: Switchover s8 master (db2165 -> db2161) from Open to In Progress.Nov 13 2025, 6:34 AM

Upgrading db2165.codfw.wmnet

Completed depool of db2165 - Upgrading db2165.codfw.wmnet - fceratto@cumin1003

Change #1204746 had a related patch set uploaded (by Federico Ceratto; author: Federico Ceratto):

[operations/puppet@production] db2165: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1204746

gerritbot added a project: Patch-For-Review.Nov 13 2025, 7:05 AM

Change #1204746 merged by Federico Ceratto:

[operations/puppet@production] db2165: Migration to MariaDB 10.11

https://gerrit.wikimedia.org/r/1204746

FCeratto-WMF updated the task description. (Show Details)Nov 13 2025, 7:12 AM