@joanna_borun as we agreed, I sent an email to cloud-announce with the following text:

Starting next month, we are going to upgrade all the Wiki Replicas [0] servers from MariaDB 10.6 to MariaDB 10.11.

We don't expect any impact to users, as there are no big breaking changes in the new version. If you are curious, you can check the full list of changes on the MariaDB website. [1]

I will send an update before we start the upgrade, you can also follow the upgrade on Phabricator. [2]

[0] https://wikitech.wikimedia.org/wiki/Help:Wiki_Replicas
[1] https://mariadb.com/kb/en/changes-improvements-in-mariadb-1011/
[2] https://phabricator.wikimedia.org/T394372

The planned date for upgrading the first host is 2025-06-09. My rough plan is:

• upgrade one web host (two sections) and check that nothing strange happens
• depool the corresponding analytics host and send all traffic for those sections to the new host
• wait one day, then repeat for all other web hosts
• upgrade the analytics hosts the following week

@Marostegui is there any page on wikitech with the procedure that you usually follow for major-version upgrades?

I was thinking of following the standard reboot procedure, but instead of running the sre.hosts.reimage cookbook, I would run the following commands:

apt remove wmf-mariadb106
apt install wmf-mariadb1011
systemctl start mariadb@sX
systemctl start mariadb@sY
mariadb-upgrade

Looping Data Platform with a tag addition here for tracking.

@dr0ptp4kt thanks! We might even start doing this for an-redacteddb1001, before clouddbs. @BTullis wdyt?

In T394372#10835232, @fnegri wrote:

@Marostegui is there any page on wikitech with the procedure that you usually follow for major-version upgrades?

I was thinking of following the standard reboot procedure, but instead of running the sre.hosts.reimage cookbook, I would run the following commands:

apt remove wmf-mariadb106
apt install wmf-mariadb1011
systemctl start mariadb@sX
systemctl start mariadb@sY
mariadb-upgrade

You'd be corrupting data this way, you first need to stop mariadb. Also, you'd need to add the key in the yaml file and finally start slave, as you're using skip-slave-start, so the process should look like:

Add mariadb::package: 'wmf-mariadb1011' in puppet and merge

systemctl stop mariadb
apt remove --purge wmf-mariadb106
run-puppet-agent
systemctl start mariadb@sX
systemctl start mariadb@sY
mariadb-upgrade
mysql -S $SOCKET_PATH -e "start slave"

you first need to stop mariadb

Sorry my message was not clear enough, I was implying these steps were after all the steps listed in the reboot procedure, so the full steps (including also your suggestions above) would be:

• Depool the host
• Downtime the host in icinga
• stop replication (both sections)
• Stop mariadb (both sections)
• apt remove --purge wmf-mariadb106
• run-puppet-agent (will this automatically install 1011? do I need a puppet patch?)
• Start mariadb (both sections)
• mariadb-upgrade (both sections)
• Start slave (both sections)
• Repool the host

run-puppet-agent (will this automatically install 1011? do I need a puppet patch?)

Ok, I didn't see your note above

Add mariadb::package: 'wmf-mariadb1011' in puppet and merge

Where is that in Puppet?

Change #1148297 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] clouddb: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1148297

Where is that in Puppet?

Found it: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1148297

In T394372#10838374, @fnegri wrote:

Where is that in Puppet?

Found it: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1148297

Yes, that's it!

@JAllemandou any implications for the Data Platform / sqooping?

I would suggest splitting the an-redacteddb1001 upgrade to a separate task. It can be a sub-task of this one, though it doesn't strictly need to happen before this one, I'm also fine with starting the upgrade on clouddbs first.

In T394372#10840706, @Ahoelzl wrote:

@JAllemandou any implications for the Data Platform / sqooping?

I expect it to be NoOp, but we better watch :)

In T394372#10844267, @fnegri wrote:

I would suggest splitting the an-redacteddb1001 upgrade to a separate task. It can be a sub-task of this one, though it doesn't strictly need to happen before this one, I'm also fine with starting the upgrade on clouddbs first.

As you prefer. If better for you, you can start by the an-redacteddb1001 :) We have very few daily jobs, mostly monthly ones. So if you wish you can try on this one next week, we verify that our monthly jobs work at the beginning of June, and then go with the rest.
The other way (other cloud-dbs first) works too :)

As you prefer. If better for you, you can start by the an-redacteddb1001 :)

No preference really, if it's upgraded first it's a good testbed, but we have 2 clouddbs per each section, so in the worst case I can just depool one.

I have created a subtask: T394930: an-redacteddb1001: upgrade MariaDB to 10.11

I will let you decide when is best to upgrade that server, and who wants to do it. I'm happy to help if you need, but I'd rather have someone from Data-Platform-SRE run the actual commands.

We are going to start with s6

We are going to start with s6

Ok, then I will start with clouddb1015 and clouddb1019 which are the two clouddbs with s6.

I added a checklist to the task description to keep track of which hosts have been upgraded, and the procedure I'm going to follow.

I plan to start working on this next Monday, June, 9th.

The steps look good to me, thank you

Change #1154803 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] clouddb1019: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1154803

Change #1154804 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] clouddb1013: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1154804

Change #1154805 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] clouddb1017: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1154805

Change #1154806 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] clouddb1014: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1154806

Change #1154807 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] clouddb1018: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1154807

Change #1154808 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] clouddb1016: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1154808

Change #1154809 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] clouddb1020: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1154809

Mentioned in SAL (#wikimedia-operations) [2025-06-09T13:52:15Z] <fnegri@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1015.eqiad.wmnet with reason: Upgrading clouddbs T394372

Change #1148297 merged by FNegri:

[operations/puppet@production] clouddb1015: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1148297

clouddb1015 is upgraded and repooled! I will continue with clouddb1019 tomorrow.

Mentioned in SAL (#wikimedia-operations) [2025-06-10T09:26:58Z] <fnegri@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1019.eqiad.wmnet with reason: Upgrading clouddbs T394372

Change #1154803 merged by FNegri:

[operations/puppet@production] clouddb1019: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1154803

clouddb1019 upgraded and repooled.

Note: it looks like the new MariaDB version errors out on some selects involving assigments (:=) that previously worked. Reading changelogs, release notes and docs, don't understand why. Spotted in T396423; now circumvented there by not using assigments, but there's probably other code out there that got broken.

@Alien333 thanks for reporting this!

I don't think the assignments (:=) are the problem, I narrowed it down to MariaDB 10.11 complaining if you use AS row_number without quoting it. Given that row_number is a reserved word, MariaDB 10.11 complains unless you quote it with backticks:

AS `row_number`

I tried to find exactly when this changed, but I could not find a related commit or Jira issue in https://jira.mariadb.org/.

I can reproduce this with a minimal query:

# Works on 10.6 but fails on 10.11
SELECT 1 AS row_number;

# Works on both 10.6 and 10.11
SELECT 1 AS `row_number`;

From MariaDB 10.7 according to https://mariadb.com/kb/en/reserved-words/ ;)

From MariaDB 10.7 according to https://mariadb.com/kb/en/reserved-words/ ;)

That's some good sleuthing, I spent at least 30 minutes looking for that and you found it in 2 mins! :D

Thank you very much for the precision.

Mentioned in SAL (#wikimedia-operations) [2025-06-16T15:20:35Z] <fnegri@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1013.eqiad.wmnet with reason: Upgrading clouddbs T394372

Change #1154804 merged by FNegri:

[operations/puppet@production] clouddb1013: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1154804

clouddb1013 upgraded and repooled.

I have now depooled clouddb1017 so all traffic for s1 and s3 is going to the just-upgraded clouddb1013 host.

Given that I anticipate more wikireplicas queries for s1 than for other sections, I will wait until tomorrow before upgrading 1017, just in case some unexpected issue appears on 1013 and we need more time to investigate.

Note: I have temporarily increased the wmf-pt-kill timeout in clouddb1013 to match the (higher) value of 10800 found in clouddb1017. I will revert this when clouddb1017 is upgraded and repooled.

Mentioned in SAL (#wikimedia-operations) [2025-06-17T09:30:41Z] <fnegri@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1017.eqiad.wmnet with reason: Upgrading clouddbs T394372

Change #1154805 merged by FNegri:

[operations/puppet@production] clouddb1017: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1154805

clouddb1017 upgraded and repooled.

I restored the default wmt-pt-kill timeout in clouddb1013.

Mentioned in SAL (#wikimedia-operations) [2025-06-17T10:10:58Z] <fnegri@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1014.eqiad.wmnet with reason: Upgrading clouddbs T394372

Change #1154806 merged by FNegri:

[operations/puppet@production] clouddb1014: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1154806

clouddb1014 upgraded and repooled.

Mentioned in SAL (#wikimedia-operations) [2025-06-17T10:30:23Z] <fnegri@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1018.eqiad.wmnet with reason: Upgrading clouddbs T394372

Change #1154807 merged by FNegri:

[operations/puppet@production] clouddb1018: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1154807

clouddb1018 upgraded and repooled.

Mentioned in SAL (#wikimedia-operations) [2025-06-18T08:42:16Z] <fnegri@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1016.eqiad.wmnet with reason: Upgrading clouddbs T394372

Change #1154808 merged by FNegri:

[operations/puppet@production] clouddb1016: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1154808

Change #1154809 merged by FNegri:

[operations/puppet@production] clouddb1020: upgrade to MariaDB 10.11

https://gerrit.wikimedia.org/r/1154809

Mentioned in SAL (#wikimedia-operations) [2025-06-18T09:18:53Z] <fnegri@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1020.eqiad.wmnet with reason: Upgrading clouddbs T394372

Thank you!