Page MenuHomePhabricator
Create Task
Maniphest T334650

Migrate s6 to MariaDB 10.6
Closed, ResolvedPublic
Actions

Description

  • dbstore1005.eqiad.wmnet T334652
  • db2187.codfw.wmnet
  • db2180.codfw.wmnet
  • db2171.codfw.wmnet
  • db2169.codfw.wmnet
  • db2158.codfw.wmnet
  • db2151.codfw.wmnet
  • db2129.codfw.wmnet
  • db2124.codfw.wmnet
  • db2117.codfw.wmnet
  • db2097.codfw.wmnet (was: db2141.codfw.wmnet)
  • db2114.codfw.wmnet
  • db1224.eqiad.wmnet
  • db1201.eqiad.wmnet
  • db1187.eqiad.wmnet
  • db1180.eqiad.wmnet
  • db1173.eqiad.wmnet
  • db1168.eqiad.wmnet
  • db1165.eqiad.wmnet
  • db1155.eqiad.wmnet
  • db1225.eqiad.wmnet (was db1140.eqiad.wmnet)
  • db1131.eqiad.wmnet
  • db1213.eqiad.wmnet
  • clouddb1021.eqiad.wmnet T334651
  • clouddb1019.eqiad.wmnet T334651
  • clouddb1015.eqiad.wmnet T334651

Details

SubjectRepoBranchLines +/-
mariadb: Upgrade db1225 to mariadb 10.6 (and generate 10.6 backups)operations/puppetproduction+3 -1
db1155: Upgrade to mariadb 10.6operations/puppetproduction+1 -0
dbproxy1018: Depool clouddb1019operations/puppetproduction+7 -7
dbproxy1018: Depool clouddb1019operations/puppetproduction+6 -6
dbproxy1019: Depool clouddb1015operations/puppetproduction+7 -7
clouddb1015: Migrate to MariaDB 10.6operations/puppetproduction+1 -0
db1224.yaml: Migrate to MariaDB 10.6operations/puppetproduction+1 -0
db2129: Migrate to 10.6operations/puppetproduction+2 -1
db2114: Migrate to MariaDB 10.6operations/puppetproduction+2 -1
mariadb: Reenable notifications for db1140, db1225operations/puppetproduction+0 -2
mariadb: Switch s3 and s6 from db1140, db1225operations/puppetproduction+6 -7
mariadb: Disable notifications on db1140, db1225 for maintenanceoperations/puppetproduction+2 -0
mariadb: Enable notifications for db2097, db2141operations/puppetproduction+0 -2
mariadb: Upgrade db2097 to use mariadb 10.6 packageoperations/puppetproduction+2 -1
mariadb: Disable notifications for db2097, db2141operations/puppetproduction+2 -0
mariadb: Move s6 from db2141 to db2097 and drop s1 & add x1operations/puppetproduction+8 -8
dbbackups: Reorganize backups so dbprov1004,2006 create 10.6 onesoperations/puppetproduction+28 -27
dbbackups: Upgrade dbprov1004,2004 to MariaDB 10.6operations/puppetproduction+2 -0
db1213: Migrate to MariaDB 10.6operations/puppetproduction+1 -0
db1187: Migrate to mariadb 10.6operations/puppetproduction+1 -0
db2169: Migrate to MariaDB 10.6operations/puppetproduction+1 -0
db1201: Migrate to mariadb 10.6operations/puppetproduction+1 -0
db2171: Migrate to mariadb 10.6operations/puppetproduction+1 -0
db1180: Migrate to mariadb 10.6operations/puppetproduction+1 -0
db2158: Migrate to MariaDB 10.6operations/puppetproduction+1 -0
db2187: Migrate to 10.6operations/puppetproduction+1 -0
db2117: Migrate to 10.6operations/puppetproduction+1 -0
db2151: Migrat to 10.6operations/puppetproduction+1 -0
db2124: Migrate to 10.6operations/puppetproduction+2 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Maintenance_bot removed a project: Patch-For-Review.Jul 21 2023, 7:31 AM
gerritbot added a comment.Jul 24 2023, 12:06 PM

Change 940934 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2169: Migrate to MariaDB 10.6

https://gerrit.wikimedia.org/r/940934

gerritbot added a project: Patch-For-Review.Jul 24 2023, 12:06 PM

Change 940934 merged by Marostegui:

[operations/puppet@production] db2169: Migrate to MariaDB 10.6

https://gerrit.wikimedia.org/r/940934

• Marostegui updated the task description. (Show Details)Jul 24 2023, 12:10 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 24 2023, 12:11 PM
gerritbot added a comment.Jul 24 2023, 12:14 PM

Change 940935 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1187: Migrate to mariadb 10.6

https://gerrit.wikimedia.org/r/940935

gerritbot added a project: Patch-For-Review.Jul 24 2023, 12:14 PM

Change 940935 merged by Marostegui:

[operations/puppet@production] db1187: Migrate to mariadb 10.6

https://gerrit.wikimedia.org/r/940935

• Marostegui updated the task description. (Show Details)Jul 24 2023, 12:15 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 24 2023, 12:31 PM
gerritbot added a comment.Jul 25 2023, 6:12 AM

Change 941040 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1213: Migrate to MariaDB 10.6

https://gerrit.wikimedia.org/r/941040

gerritbot added a project: Patch-For-Review.Jul 25 2023, 6:12 AM
gerritbot added a comment.Jul 25 2023, 6:14 AM

Change 941040 merged by Marostegui:

[operations/puppet@production] db1213: Migrate to MariaDB 10.6

https://gerrit.wikimedia.org/r/941040

• Marostegui updated the task description. (Show Details)Jul 25 2023, 6:17 AM
Maintenance_bot removed a project: Patch-For-Review.Jul 25 2023, 6:30 AM
• Marostegui updated the task description. (Show Details)Jul 26 2023, 6:51 AM
• Marostegui added a comment.Jul 27 2023, 7:01 AM

@jcrespo how would you like to proceed with the backup sources in codfw? I can do it, as soon as you give me green light for it (it is db2141 and it also has s1)

• jcrespo added a comment.Jul 27 2023, 9:23 AM

I need to make sure the dbprov host also has 10.6- should be done today.

gerritbot added a comment.Jul 27 2023, 9:40 AM

Change 942376 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Upgrade dbprov1004 to MariaDB 10.6

https://gerrit.wikimedia.org/r/942376

gerritbot added a project: Patch-For-Review.Jul 27 2023, 9:40 AM
• Marostegui added a comment.Jul 27 2023, 9:50 AM

Ah right - no rush. Let me know when done so I can go ahead and do the backup source + candidate master

gerritbot added a comment.Jul 27 2023, 10:02 AM

Change 942380 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Reorganize backups so dbprov1004,2006 creates 10.6 ones

https://gerrit.wikimedia.org/r/942380

gerritbot added a comment.Jul 27 2023, 10:19 AM

Change 942376 merged by Jcrespo:

[operations/puppet@production] dbbackups: Upgrade dbprov1004,2004 to MariaDB 10.6

https://gerrit.wikimedia.org/r/942376

gerritbot added a comment.Jul 27 2023, 10:27 AM

Change 942380 merged by Jcrespo:

[operations/puppet@production] dbbackups: Reorganize backups so dbprov1004,2006 create 10.6 ones

https://gerrit.wikimedia.org/r/942380

Maintenance_bot removed a project: Patch-For-Review.Jul 27 2023, 10:29 AM
• jcrespo added a comment.Jul 27 2023, 11:34 AM

Here is the plan: swap s3 and s6 db1225 (s2, s3, x1) and db1140 (s1, s6) so they end up like this:

  • db1225 (s2, s6, x1)
  • db1140 (s1, s3)

Then upgrade db1225 to s6

and: move s6 from db2141 (s1, s6) to db2097 (s1, s2) then drop s1 from db2097 and add x1, ending up like this:

  • db2141 (s1)
  • db2097 (s2, s6, x1)

Then upgrade db2097.

I will do the second first.

gerritbot added a comment.Jul 27 2023, 11:48 AM

Change 942415 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Disable notifications for db2097, db2141

https://gerrit.wikimedia.org/r/942415

gerritbot added a project: Patch-For-Review.Jul 27 2023, 11:48 AM

Change 942416 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Move s6 from db2141 to db2097 and drop s1 & add x1

https://gerrit.wikimedia.org/r/942416

gerritbot added a comment.Jul 27 2023, 11:49 AM

Change 942415 merged by Jcrespo:

[operations/puppet@production] mariadb: Disable notifications for db2097, db2141

https://gerrit.wikimedia.org/r/942415

gerritbot added a comment.Jul 27 2023, 12:44 PM

Change 942416 merged by Jcrespo:

[operations/puppet@production] mariadb: Move s6 from db2141 to db2097 and drop s1 & add x1

https://gerrit.wikimedia.org/r/942416

Maintenance_bot removed a project: Patch-For-Review.Jul 27 2023, 1:12 PM
• jcrespo added a comment.EditedJul 27 2023, 4:06 PM

@Marostegui I have prepared db2097 to contain s6 (as well as s2 and x1, the probably next servers to upgrade) with 10.4, to backup to dbprov2004 (10.6).

We can now upgrade db2097 to 10.6 on bullseye unless you have any comment or thing to have into account regarding the upgrade procedures. If not, I can do it on my own (feel free to do it fully on your own if you see this early and I am not around, there are no backup running tomorrow morning).

Once that is finished we can start generating 10.6 snapshots and proceed with the eqiad changes.

Note: db2141 has removed its s6 instance (changes updated on zarcillo)

• jcrespo updated the task description. (Show Details)Jul 27 2023, 4:07 PM
gerritbot added a comment.Jul 27 2023, 5:06 PM

Change 942461 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Upgrade db2097 to use mariadb 10.6 package

https://gerrit.wikimedia.org/r/942461

gerritbot added a project: Patch-For-Review.Jul 27 2023, 5:06 PM
• Marostegui added a comment.Jul 27 2023, 5:17 PM

Great! I'll upgrade 2097 then. Thanks a lot for getting this moved around so fast

gerritbot added a comment.Jul 28 2023, 5:33 AM

Change 942461 merged by Marostegui:

[operations/puppet@production] mariadb: Upgrade db2097 to use mariadb 10.6 package

https://gerrit.wikimedia.org/r/942461

gerritbot added a comment.Jul 28 2023, 5:38 AM

Change 942537 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Enable notifications for db2097, db2141

https://gerrit.wikimedia.org/r/942537

gerritbot added a comment.Jul 28 2023, 5:39 AM

Change 942537 merged by Marostegui:

[operations/puppet@production] mariadb: Enable notifications for db2097, db2141

https://gerrit.wikimedia.org/r/942537

• Marostegui updated the task description. (Show Details)Jul 28 2023, 5:40 AM
Maintenance_bot removed a project: Patch-For-Review.Jul 28 2023, 6:10 AM
• Marostegui added a subtask: T342947: Switchover s6 master (db2129 -> db2114).Jul 28 2023, 6:32 AM
• Marostegui changed the status of subtask T342947: Switchover s6 master (db2129 -> db2114) from Open to Stalled.
• jcrespo added a comment.Jul 28 2023, 7:29 AM

Any ETA, even if vague for the eqiad upgrade?

• Marostegui added a comment.Jul 28 2023, 11:04 AM
In T334650#9050440, @jcrespo wrote:

Any ETA, even if vague for the eqiad upgrade?

Hopefully next week I want to have all done except the candidate and the primary

gerritbot added a comment.Jul 28 2023, 11:15 AM

Change 942636 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Switch s3 and s6 from db1140, db1225

https://gerrit.wikimedia.org/r/942636

gerritbot added a project: Patch-For-Review.Jul 28 2023, 11:15 AM
gerritbot added a comment.Jul 28 2023, 11:17 AM

Change 942637 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Disable notifications on db1140, db1225 for maintenance

https://gerrit.wikimedia.org/r/942637

gerritbot added a comment.Jul 28 2023, 11:19 AM

Change 942637 merged by Jcrespo:

[operations/puppet@production] mariadb: Disable notifications on db1140, db1225 for maintenance

https://gerrit.wikimedia.org/r/942637

gerritbot added a comment.Jul 28 2023, 11:34 AM

Change 942636 merged by Jcrespo:

[operations/puppet@production] mariadb: Switch s3 and s6 from db1140, db1225

https://gerrit.wikimedia.org/r/942636

• jcrespo updated the task description. (Show Details)Jul 28 2023, 12:05 PM

db1225.eqiad.wmnet is now the place where s6 backups are taken (it also contains x1 and s2), and it is ready to be upgraded. I will send the patches to do so, but it is only blocked on finishing the migration (just before the eqiad primary upgrade, with no other blocker).

• Marostegui added a comment.Jul 28 2023, 12:07 PM

Excellent! I will upgrade it on Monday. Thanks a lot!

Maintenance_bot removed a project: Patch-For-Review.Jul 28 2023, 12:10 PM
gerritbot added a comment.Jul 28 2023, 12:15 PM

Change 942645 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Reenable notifications for db1140, db1225

https://gerrit.wikimedia.org/r/942645

gerritbot added a project: Patch-For-Review.Jul 28 2023, 12:15 PM
gerritbot added a comment.Jul 28 2023, 12:50 PM

Change 942645 merged by Jcrespo:

[operations/puppet@production] mariadb: Reenable notifications for db1140, db1225

https://gerrit.wikimedia.org/r/942645

gerritbot added a comment.Jul 28 2023, 1:09 PM

Change 942652 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Upgrade db1225 to mariadb 10.6 (and generate 10.6 backups)

https://gerrit.wikimedia.org/r/942652

• jcrespo added a comment.Jul 28 2023, 2:43 PM

Not enough data points, but 10.6 backup is 10 minutes faster than 10.4:

Screenshot_20230728_164018.png (257×1 px, 68 KB)

Stashbot added a comment.Jul 31 2023, 11:46 AM

Mentioned in SAL (#wikimedia-operations) [2023-07-31T11:46:45Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2114 T334650', diff saved to https://phabricator.wikimedia.org/P49803 and previous config saved to /var/cache/conftool/dbconfig/20230731-114645-root.json

gerritbot added a comment.Jul 31 2023, 11:47 AM

Change 943550 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2114: Migrate to MariaDB 10.6

https://gerrit.wikimedia.org/r/943550

gerritbot added a comment.Jul 31 2023, 11:48 AM

Change 943550 merged by Marostegui:

[operations/puppet@production] db2114: Migrate to MariaDB 10.6

https://gerrit.wikimedia.org/r/943550

• Marostegui updated the task description. (Show Details)Jul 31 2023, 11:51 AM
• Marostegui added a subtask: T343296: Switchover s6 master (db2129 -> db2114).Aug 2 2023, 6:31 AM
• Marostegui closed subtask T343296: Switchover s6 master (db2129 -> db2114) as Resolved.Aug 3 2023, 6:04 AM
gerritbot added a comment.Aug 3 2023, 6:13 AM

Change 945462 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2129: Migrate to 10.6

https://gerrit.wikimedia.org/r/945462

gerritbot added a comment.Aug 3 2023, 6:22 AM

Change 945462 merged by Marostegui:

[operations/puppet@production] db2129: Migrate to 10.6

https://gerrit.wikimedia.org/r/945462

• Marostegui updated the task description. (Show Details)Aug 3 2023, 6:22 AM
gerritbot added a comment.Aug 7 2023, 6:18 AM

Change 946342 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1224.yaml: Migrate to MariaDB 10.6

https://gerrit.wikimedia.org/r/946342

gerritbot added a comment.Aug 7 2023, 6:22 AM

Change 946342 merged by Marostegui:

[operations/puppet@production] db1224.yaml: Migrate to MariaDB 10.6

https://gerrit.wikimedia.org/r/946342

• Marostegui updated the task description. (Show Details)Aug 7 2023, 6:24 AM
gerritbot added a comment.Aug 7 2023, 6:28 AM

Change 946343 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1019: Depool clouddb1015

https://gerrit.wikimedia.org/r/946343

gerritbot added a comment.Aug 7 2023, 7:11 AM

Change 946343 merged by Marostegui:

[operations/puppet@production] dbproxy1019: Depool clouddb1015

https://gerrit.wikimedia.org/r/946343

Stashbot added a comment.Aug 7 2023, 7:11 AM

Mentioned in SAL (#wikimedia-operations) [2023-08-07T07:11:34Z] <marostegui> Depool clouddb1015 T334650

gerritbot added a comment.Aug 7 2023, 7:13 AM

Change 946507 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] clouddb1015: Migrate to MariaDB 10.6

https://gerrit.wikimedia.org/r/946507

gerritbot added a comment.Aug 7 2023, 7:13 AM

Change 946507 merged by Marostegui:

[operations/puppet@production] clouddb1015: Migrate to MariaDB 10.6

https://gerrit.wikimedia.org/r/946507

• Marostegui updated the task description. (Show Details)Aug 7 2023, 7:17 AM
gerritbot added a comment.Aug 21 2023, 6:05 AM

Change 950964 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1018: Depool clouddb1019

https://gerrit.wikimedia.org/r/950964

gerritbot added a comment.Aug 21 2023, 6:06 AM

Change 950964 abandoned by Marostegui:

[operations/puppet@production] dbproxy1018: Depool clouddb1019

Reason:

wrong branch

https://gerrit.wikimedia.org/r/950964

gerritbot added a comment.Aug 21 2023, 6:07 AM

Change 950965 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1018: Depool clouddb1019

https://gerrit.wikimedia.org/r/950965

gerritbot added a comment.Aug 21 2023, 6:15 AM

Change 950965 merged by Marostegui:

[operations/puppet@production] dbproxy1018: Depool clouddb1019

https://gerrit.wikimedia.org/r/950965

• Marostegui closed subtask T334651: Migrate wiki replicas (clouddb*) hosts to MariaDB 10.6 as Resolved.Aug 21 2023, 6:22 AM
• Marostegui updated the task description. (Show Details)
gerritbot added a comment.Aug 29 2023, 5:38 AM

Change 953133 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1155: Upgrade to mariadb 10.6

https://gerrit.wikimedia.org/r/953133

gerritbot added a comment.Aug 29 2023, 5:40 AM

Change 953133 merged by Marostegui:

[operations/puppet@production] db1155: Upgrade to mariadb 10.6

https://gerrit.wikimedia.org/r/953133

• Marostegui updated the task description. (Show Details)Aug 29 2023, 5:43 AM
• Marostegui updated the task description. (Show Details)Aug 30 2023, 5:48 AM
gerritbot added a comment.Aug 30 2023, 5:52 AM

Change 942652 merged by Marostegui:

[operations/puppet@production] mariadb: Upgrade db1225 to mariadb 10.6 (and generate 10.6 backups)

https://gerrit.wikimedia.org/r/942652

• Marostegui updated the task description. (Show Details)Aug 30 2023, 5:55 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 30 2023, 6:10 AM
• Marostegui added a subtask: T345223: Switchover s6 master (db1131 -> db1173).Aug 31 2023, 5:36 AM
• Marostegui closed this task as Resolved.Aug 31 2023, 7:10 AM
• Marostegui updated the task description. (Show Details)

This is all done

Maintenance_bot moved this task from In progress to Done on the DBA board.Aug 31 2023, 7:15 AM
• jcrespo awarded a token.Aug 31 2023, 7:24 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits