Page MenuHomePhabricator
Create Task
Maniphest T399540

Upgrade masters to 10.6.22 and 10.11.13 .2 update
Closed, ResolvedPublic
Actions

Description

At T397425: Build 10.6.22 and 10.11.13 with mdev36934 patch we had to patch our current 10.6.22 and 10.11.13 with an update as we were bitten by https://jira.mariadb.org/browse/MDEV-36934
Today for T399446: Switchover s5 master (db1230 -> db1210) as the primary wasn't patched we got bitten and the switchover had to be finished manually, as it failed when inverting replication.

Masters progress:

  • db1151.eqiad.wmnet
  • db1152.eqiad.wmnet
  • db1153.eqiad.wmnet
  • db1163.eqiad.wmnet
  • db1176.eqiad.wmnet (test host)
  • db1193.eqiad.wmnet
  • db1201.eqiad.wmnet
  • db1204.eqiad.wmnet
  • db1210.eqiad.wmnet
  • db1213.eqiad.wmnet
  • db1215.eqiad.wmnet (zarcillo)
  • db1220.eqiad.wmnet
  • db1222.eqiad.wmnet
  • db1223.eqiad.wmnet
  • db1228.eqiad.wmnet
  • db1236.eqiad.wmnet
  • db1244.eqiad.wmnet
  • db1250.eqiad.wmnet
  • db1255.eqiad.wmnet
  • es1035.eqiad.wmnet
  • es1038.eqiad.wmnet
  • pc1011.eqiad.wmnet
  • pc1012.eqiad.wmnet
  • pc1013.eqiad.wmnet
  • pc1014.eqiad.wmnet
  • pc1015.eqiad.wmnet
  • pc1016.eqiad.wmnet
  • pc1017.eqiad.wmnet
  • pc1018.eqiad.wmnet
  • db2142.codfw.wmnet
  • db2143.codfw.wmnet
  • db2144.codfw.wmnet
  • db2151.codfw.wmnet
  • db2165.codfw.wmnet
  • db2179.codfw.wmnet
  • db2183.codfw.wmnet
  • db2196.codfw.wmnet
  • db2203.codfw.wmnet
  • db2204.codfw.wmnet
  • db2209.codfw.wmnet
  • db2213.codfw.wmnet
  • db2214.codfw.wmnet
  • db2218.codfw.wmnet
  • db2232.codfw.wmnet
  • db2233.codfw.wmnet
  • db2234.codfw.wmnet
  • db2235.codfw.wmnet
  • db2241.codfw.wmnet
  • es2037.codfw.wmnet
  • es2038.codfw.wmnet
  • pc2011.codfw.wmnet
  • pc2012.codfw.wmnet
  • pc2013.codfw.wmnet
  • pc2014.codfw.wmnet
  • pc2015.codfw.wmnet
  • pc2016.codfw.wmnet
  • pc2017.codfw.wmnet
  • pc2018.codfw.wmnet

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• Marostegui closed subtask T400027: Switchover es6 master (es1038 -> es1037) as Declined.Jul 21 2025, 7:18 AM
• Marostegui updated the task description. (Show Details)Jul 21 2025, 7:40 AM
• Marostegui updated the task description. (Show Details)Jul 21 2025, 8:06 AM
• Marostegui closed subtask T399930: Switchover s3 master (db2209 -> db2205) as Resolved.Jul 21 2025, 8:11 AM
• Marostegui updated the task description. (Show Details)Jul 25 2025, 9:10 AM
• Marostegui updated the task description. (Show Details)Jul 28 2025, 7:53 AM
ops-monitoring-bot added a comment.Aug 13 2025, 1:12 PM

Upgrading db2165.codfw.wmnet

ops-monitoring-bot added a comment.Aug 13 2025, 1:13 PM

Completed depool of db2165 - Upgrading db2165.codfw.wmnet - fceratto@cumin1002

ops-monitoring-bot added a comment.Aug 13 2025, 1:20 PM

Start pool of db2165 gradually with 4 steps - Upgrade of db2165.codfw.wmnet completed - fceratto@cumin1002

ops-monitoring-bot added a comment.Aug 13 2025, 2:06 PM

Completed pool of db2165 gradually with 4 steps - Upgrade of db2165.codfw.wmnet completed - fceratto@cumin1002

ops-monitoring-bot added a comment.Aug 13 2025, 2:06 PM

Upgrade of db2165.codfw.wmnet completed

FCeratto-WMF updated the task description. (Show Details)Aug 13 2025, 3:40 PM
ops-monitoring-bot added a comment.Aug 14 2025, 11:08 AM

Depooled db1152.eqiad.wmnet and db2142.codfw.wmnet Depool for MariaDB upgrade - fceratto@cumin1002 - T399540

ops-monitoring-bot added a comment.Aug 14 2025, 11:28 AM

Upgrading db2142.codfw.wmnet

ops-monitoring-bot added a comment.Aug 14 2025, 11:56 AM

Upgrading db2142.codfw.wmnet

ops-monitoring-bot added a comment.Aug 14 2025, 12:03 PM

Upgrade of db2142.codfw.wmnet completed

ops-monitoring-bot added a comment.Aug 14 2025, 12:03 PM

Upgrade of db2142.codfw.wmnet completed

ops-monitoring-bot added a comment.Aug 14 2025, 12:40 PM

Upgrading db1152.eqiad.wmnet

ops-monitoring-bot added a comment.Aug 14 2025, 12:46 PM

Upgrade of db1152.eqiad.wmnet completed

ops-monitoring-bot added a comment.Aug 14 2025, 12:47 PM

Upgrade of db1152.eqiad.wmnet completed

ops-monitoring-bot added a comment.Aug 14 2025, 12:57 PM

Depooled db1152.eqiad.wmnet and db2142.codfw.wmnet Repool after MariaDB upgrade - fceratto@cumin1002 - T399540

FCeratto-WMF updated the task description. (Show Details)Aug 14 2025, 12:59 PM
ops-monitoring-bot added a comment.Aug 14 2025, 2:03 PM

Depooled db1153.eqiad.wmnet and db2143.codfw.wmnet Depool for MariaDB upgrade - fceratto@cumin1002 - T399540

ops-monitoring-bot added a comment.Aug 14 2025, 2:04 PM

Upgrading db2143.codfw.wmnet

ops-monitoring-bot added a comment.Aug 14 2025, 2:10 PM

Upgrade of db2143.codfw.wmnet completed

ops-monitoring-bot added a comment.Aug 14 2025, 2:10 PM

Upgrade of db2143.codfw.wmnet completed

ops-monitoring-bot added a comment.Aug 14 2025, 2:16 PM

Upgrading db1153.eqiad.wmnet

ops-monitoring-bot added a comment.Aug 14 2025, 2:23 PM

Upgrade of db1153.eqiad.wmnet completed

ops-monitoring-bot added a comment.Aug 14 2025, 2:23 PM

Upgrade of db1153.eqiad.wmnet completed

ops-monitoring-bot added a comment.Aug 14 2025, 2:32 PM

Depooled db1153.eqiad.wmnet and db2143.codfw.wmnet Repool after MariaDB upgrade - fceratto@cumin1002 - T399540

ops-monitoring-bot added a comment.Aug 14 2025, 3:11 PM

Depooled db1151.eqiad.wmnet and db2144.codfw.wmnet Depool for MariaDB upgrade - fceratto@cumin1002 - T399540

ops-monitoring-bot added a comment.Aug 14 2025, 3:14 PM

Upgrading db1151.eqiad.wmnet

ops-monitoring-bot added a comment.Aug 14 2025, 3:15 PM

Upgrading db2144.codfw.wmnet

ops-monitoring-bot added a comment.Aug 14 2025, 3:21 PM

Upgrade of db1151.eqiad.wmnet completed

ops-monitoring-bot added a comment.Aug 14 2025, 3:21 PM

Upgrade of db1151.eqiad.wmnet completed

ops-monitoring-bot added a comment.Aug 14 2025, 3:22 PM

Upgrade of db2144.codfw.wmnet completed

ops-monitoring-bot added a comment.Aug 14 2025, 3:22 PM

Upgrade of db2144.codfw.wmnet completed

ops-monitoring-bot added a comment.Aug 14 2025, 3:55 PM

Depooled db1151.eqiad.wmnet and db2144.codfw.wmnet Depool for MariaDB upgrade - fceratto@cumin1002 - T399540

FCeratto-WMF updated the task description. (Show Details)Aug 14 2025, 4:00 PM
FCeratto-WMF subscribed.Aug 18 2025, 8:01 AM

db1176 is at version 10.6.22
db2183 has 10.11.13+deb12u2 installed and was rebooted 18 days ago
db2204 has 10.11.13+deb12u1 installed so probably needs update

ops-monitoring-bot added a comment.Aug 18 2025, 12:15 PM

Upgrading db2179.codfw.wmnet

ops-monitoring-bot added a comment.Aug 18 2025, 12:16 PM

Completed depool of db2179 - Upgrading db2179.codfw.wmnet - fceratto@cumin1002

ops-monitoring-bot added a comment.Aug 18 2025, 12:23 PM

Upgrade of db2179.codfw.wmnet completed

ops-monitoring-bot added a comment.Aug 18 2025, 12:23 PM

Upgrade of db2179.codfw.wmnet completed

ops-monitoring-bot added a comment.Aug 18 2025, 12:39 PM

Start pool of db2179* gradually with 4 steps - Upgrade MariaDB - fceratto@cumin1002

ops-monitoring-bot added a comment.Aug 18 2025, 1:24 PM

Completed pool of db2179* gradually with 4 steps - Upgrade MariaDB - fceratto@cumin1002

FCeratto-WMF updated the task description. (Show Details)Aug 18 2025, 2:40 PM
FCeratto-WMF added a subtask: T402276: Switchover s2 master (db2204 -> db2207).Aug 19 2025, 10:43 AM
ops-monitoring-bot added a comment.Aug 19 2025, 10:45 AM

Upgrading db2204.codfw.wmnet

ops-monitoring-bot added a comment.Aug 19 2025, 10:45 AM

Completed depool of db2204 - Upgrading db2204.codfw.wmnet - fceratto@cumin1002

ops-monitoring-bot added a comment.Aug 19 2025, 10:51 AM

Upgrade of db2204.codfw.wmnet completed

ops-monitoring-bot added a comment.Aug 19 2025, 10:51 AM

Upgrade of db2204.codfw.wmnet completed

ops-monitoring-bot added a comment.Aug 19 2025, 11:00 AM

Start pool of db2204* gradually with 4 steps - Upgraded MariaDB - fceratto@cumin1002

ops-monitoring-bot added a comment.Aug 19 2025, 11:45 AM

Completed pool of db2204* gradually with 4 steps - Upgraded MariaDB - fceratto@cumin1002

FCeratto-WMF updated the task description. (Show Details)Aug 19 2025, 12:55 PM
Ladsgroup updated the task description. (Show Details)Aug 26 2025, 12:17 AM
Ladsgroup updated the task description. (Show Details)Sep 2 2025, 12:44 PM
Ladsgroup subscribed.Sep 8 2025, 9:59 PM

Masters left: s3, s7, s8 in eqiad. A couple of backup sections in eqiad and codfw and db_inventory plus test-s4

Ladsgroup updated the task description. (Show Details)Sep 8 2025, 10:38 PM
Ladsgroup closed subtask T399954: Switchover s3 master (db1223 -> db1189) as Declined.
Ladsgroup updated the task description. (Show Details)Sep 8 2025, 11:09 PM
ops-monitoring-bot added a comment.Sep 9 2025, 10:48 AM

Started cloning db1172.eqiad.wmnet to db1193.eqiad.wmnet - ladsgroup@cumin1002

Ladsgroup added a comment.Sep 9 2025, 11:12 AM

Like the stupid person I am, I didn't check whether the candidate masters have the new version or not. Probably another round is needed in cases where the new master still is not on newer version.

ops-monitoring-bot added a comment.Sep 9 2025, 12:12 PM

Start pool of db1172 gradually with 4 steps - Pool db1172.eqiad.wmnet in after cloning - ladsgroup@cumin1002

Ladsgroup added a comment.Sep 9 2025, 12:24 PM
In T399540#11162088, @Ladsgroup wrote:

Like the stupid person I am, I didn't check whether the candidate masters have the new version or not. Probably another round is needed in cases where the new master still is not on newer version.

s3 is okay but s1, s4 and s8 needs another round of switchovers.

Ladsgroup updated the task description. (Show Details)Sep 9 2025, 12:45 PM
Ladsgroup updated the task description. (Show Details)
ops-monitoring-bot added a comment.Sep 9 2025, 12:58 PM

Completed pool of db1172 gradually with 4 steps - Pool db1172.eqiad.wmnet in after cloning - ladsgroup@cumin1002

Ladsgroup added a comment.EditedSep 9 2025, 1:17 PM

So these sections are needed:

  • s1
    • Upgrading the candidate master
    • Switching over
  • s4
    • Upgrading the candidate master
    • Switching over
  • s7
    • Upgrading the candidate master
    • Switching over
  • s8
    • Upgrading the candidate master
    • Switching over

I edit this comment.

ops-monitoring-bot added a comment.Sep 9 2025, 1:58 PM

Start pool of db1193 gradually with 4 steps - Pool db1193.eqiad.wmnet in after cloning - ladsgroup@cumin1002

ops-monitoring-bot added a comment.Sep 9 2025, 2:43 PM

Completed pool of db1193 gradually with 4 steps - Pool db1193.eqiad.wmnet in after cloning - ladsgroup@cumin1002

ops-monitoring-bot added a comment.Sep 9 2025, 2:43 PM

Finished cloning db1172.eqiad.wmnet to db1193.eqiad.wmnet - ladsgroup@cumin1002

FCeratto-WMF updated the task description. (Show Details)Sep 9 2025, 3:27 PM
FCeratto-WMF updated the task description. (Show Details)
ops-monitoring-bot added a comment.Sep 10 2025, 10:50 AM

Upgrading db1173.eqiad.wmnet

ops-monitoring-bot added a comment.Sep 10 2025, 10:51 AM

Completed depool of db1173 - Upgrading db1173.eqiad.wmnet - fceratto@cumin1002

ops-monitoring-bot added a comment.Sep 10 2025, 10:57 AM

Upgrade of db1173.eqiad.wmnet completed

ops-monitoring-bot added a comment.Sep 10 2025, 10:57 AM

Upgrade of db1173.eqiad.wmnet completed

ops-monitoring-bot added a comment.Sep 10 2025, 11:04 AM

Upgrading db1173.eqiad.wmnet

ops-monitoring-bot added a comment.Sep 10 2025, 11:04 AM

Completed depool of db1173 - Upgrading db1173.eqiad.wmnet - fceratto@cumin1002

ops-monitoring-bot added a comment.Sep 10 2025, 11:10 AM

Start pool of db1173 gradually with 4 steps - Upgrade of db1173.eqiad.wmnet completed - fceratto@cumin1002

ops-monitoring-bot added a comment.Sep 10 2025, 11:33 AM

Upgrading db2191.codfw.wmnet

ops-monitoring-bot added a comment.Sep 10 2025, 11:33 AM

Completed depool of db2191 - Upgrading db2191.codfw.wmnet - fceratto@cumin1002

Ladsgroup updated the task description. (Show Details)Sep 10 2025, 11:39 AM
ops-monitoring-bot added a comment.Sep 10 2025, 11:44 AM

Start pool of db2191 gradually with 4 steps - Upgrade of db2191.codfw.wmnet completed - fceratto@cumin1002

ops-monitoring-bot added a comment.Sep 10 2025, 11:56 AM

Completed pool of db1173 gradually with 4 steps - Upgrade of db1173.eqiad.wmnet completed - fceratto@cumin1002

ops-monitoring-bot added a comment.Sep 10 2025, 11:56 AM

Upgrade of db1173.eqiad.wmnet completed

ops-monitoring-bot added a comment.Sep 10 2025, 12:30 PM

Completed pool of db2191 gradually with 4 steps - Upgrade of db2191.codfw.wmnet completed - fceratto@cumin1002

ops-monitoring-bot added a comment.Sep 10 2025, 12:30 PM

Upgrade of db2191.codfw.wmnet completed

Stashbot added a comment.Sep 10 2025, 12:55 PM

Mentioned in SAL (#wikimedia-operations) [2025-09-10T12:55:02Z] <ladsgroup@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1215.eqiad.wmnet with reason: Glow up (T399540 T394371)

Stashbot mentioned this in T394371: Migrate to MariaDB 10.11.Sep 10 2025, 12:55 PM
Ladsgroup updated the task description. (Show Details)Sep 10 2025, 1:01 PM
Ladsgroup mentioned this in T404326: Switchover s1 master (db1184 -> db1163).Sep 11 2025, 10:56 AM
Ladsgroup closed this task as Resolved.Sep 11 2025, 11:22 AM
Ladsgroup moved this task from In progress to Done on the DBA board.
Ladsgroup updated the task description. (Show Details)
• Marostegui mentioned this in T406543: Compile and package MariaDB 10.11.14.Oct 7 2025, 6:00 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits