@jcrespo I have assigned this to you as the backup owner so we need to coordinate with you, but if during any of the section upgrades you want us to proceed with the backup source upgrade, just let us know.
We'll ask you in any case!

@jcrespo for your information, s2 codfw is almost done upgrading to 10.6, only pending the master.

Thanks for keeping me up to date, will rearrange the backup sources and dbprov accordingly. s6 eqiad and s5 next ?

I believe @ABran-WMF is doing s5 now, but I am not sure.

I am ! T360116 I'll still have to run a replication source switch for eqiad though

@jcrespo s2 fully done. Next is s3.

Change #1018247 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Migrate db2097 backups to db2197

https://gerrit.wikimedia.org/r/1018247

Change #1018276 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Migrate db2098 backups to db2198 and upgrade dbprov2002 to 10.6

https://gerrit.wikimedia.org/r/1018276

In T360751#9699959, @Marostegui wrote:

@jcrespo s2 fully done. Next is s3.

It seems to me that the upgrade pace has increased (good job!). Do you have a guesstimate of how many sections will be in 10.6 more or less fully (even if there is 1 or 2 hosts left) by the end of the quarter, even if aspirationally? That way I can sync by EOQ (my main blocker is how many of the 4 dbprovs to reimage/upgrade to 10.6 by end of June).

I think there will be at least 3 more (s6 is already done) so that would be:
s2 (done), s5 almost done, s3, s7 and s8

As MariaDB 10.4 is EOL in June 18th... we are sort of moving fast

Thanks, that informs me that either 2 or 3 should be upgraded by the end of the quarter, then. Thank you a lot!

Change #1018247 merged by Jcrespo:

[operations/puppet@production] mariadb: Migrate db2097 backups to db2197

https://gerrit.wikimedia.org/r/1018247

@jcrespo eqiad s3 is almost done T362134

Change #1019263 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Move services db2101->db2201,db2099->db2199, upgrade dbprov2003

https://gerrit.wikimedia.org/r/1019263

Change #1019679 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Migrate s5 codfw snapshots to dbprov2005/db2201

https://gerrit.wikimedia.org/r/1019679

Change #1019679 merged by Jcrespo:

[operations/puppet@production] dbbackups: Migrate s5 codfw snapshots to dbprov2005/db2201

https://gerrit.wikimedia.org/r/1019679

Change #1019690 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Upgrade db2139 to MariaDB 10.6

https://gerrit.wikimedia.org/r/1019690

Mentioned in SAL (#wikimedia-operations) [2024-04-15T17:13:53Z] <jynus> stop db2139 dbs for upgrade T360751

Change #1019690 merged by Jcrespo:

[operations/puppet@production] mariadb: Upgrade db2139 to MariaDB 10.6

https://gerrit.wikimedia.org/r/1019690

@Marostegui Update: backups for x1, s2, s6, s5 and s3 are generating dumps and snapshots with MariaDB 10.6 currently on codfw. Doing s5 and s3 on eqiad next. You may see a lot of 10.4 servers, but they are idle and only kept just in case, they are not active, and will be just eventually upgraded or discarded.

Preparing s7 and s8 next.

Thanks, s3 codfw is almost done. Only pending the master which I'll try to do this week

s3 codfw master switched.

@jcrespo I am starting to work with s7 codfw

Change #1020685 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadbd: Upgrade mariadb package on db1240 from 10.4 to 10.6

https://gerrit.wikimedia.org/r/1020685

Change #1020685 merged by Jcrespo:

[operations/puppet@production] mariadbd: Upgrade mariadb package on db1240 from 10.4 to 10.6

https://gerrit.wikimedia.org/r/1020685

Change #1020693 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Upgrade mariadb package on db1216 from 10.4 to 10.6

https://gerrit.wikimedia.org/r/1020693

Change #1020693 merged by Jcrespo:

[operations/puppet@production] mariadb: Upgrade mariadb package on db1216 from 10.4 to 10.6

https://gerrit.wikimedia.org/r/1020693

Change #1020697 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] site.pp: Reorder backup sources by server name and update comments

https://gerrit.wikimedia.org/r/1020697

Change #1020697 merged by Jcrespo:

[operations/puppet@production] site.pp: Reorder backup sources by server name and update comments

https://gerrit.wikimedia.org/r/1020697

update: on both eqiad and codfw we are generating dumps and snapshots in 10.6 for x1, s2, s6, s5, s3.

I am seeing that s7 and s4 are next. Are you aiming to complete all sections by the end of the quarter (not caring about the top primaries)?

yes we're aiming to that!

Change #1025710 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Upgrade db1150, db1171 and move s4, s7, s8 backups to 10.6

https://gerrit.wikimedia.org/r/1025710

Change #1018276 abandoned by Jcrespo:

[operations/puppet@production] mariadb: Migrate db2098 backups to db2198 and upgrade dbprov2002 to 10.6

Reason:

in favour of a more up to date change

https://gerrit.wikimedia.org/r/1018276

Change #1019263 abandoned by Jcrespo:

[operations/puppet@production] mariadb: Move services db2101->db2201,db2099->db2199, upgrade dbprov2003

Reason:

in favour of a more up to date change

https://gerrit.wikimedia.org/r/1019263

@jcrespo s8 codfw is done, only pending the master.

In T360751#9764148, @Marostegui wrote:

@jcrespo s8 codfw is done, only pending the master.

s8 codfw fully done

s8 eqiad is almost done

Change #1025710 merged by Jcrespo:

[operations/puppet@production] mariadb: Upgrade db1150, db1171 and move s4, s7, s8 backups to 10.6

https://gerrit.wikimedia.org/r/1025710

All backups now will be generated from 10.6 servers, with the exception of s1. Leaving a couple of hosts with 10.4 before upgrading them/decomming them.

Change #1037107 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Migrate s1 backups to MariaDB 10.6

https://gerrit.wikimedia.org/r/1037107

Change #1037107 merged by Jcrespo:

[operations/puppet@production] dbbackups: Migrate s1 backups to MariaDB 10.6

https://gerrit.wikimedia.org/r/1037107

Change #1039973 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Prepare db2098 for decommission

https://gerrit.wikimedia.org/r/1039973

Change #1039973 merged by Jcrespo:

[operations/puppet@production] dbbackups: Prepare db2098 for decommission

https://gerrit.wikimedia.org/r/1039973

Change #1039979 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Prepare for db2097 decommission

https://gerrit.wikimedia.org/r/1039979

Change #1039999 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Prepare db2099 for decommission

https://gerrit.wikimedia.org/r/1039999

Change #1039979 merged by Jcrespo:

[operations/puppet@production] dbbackups: Prepare for db2097 decommission

https://gerrit.wikimedia.org/r/1039979

Change #1039999 merged by Jcrespo:

[operations/puppet@production] dbbackups: Prepare db2099 for decommission

https://gerrit.wikimedia.org/r/1039999

Change #1040087 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Remove all puppet references for db2097, db2098, db2099

https://gerrit.wikimedia.org/r/1040087

Change #1040087 merged by Jcrespo:

[operations/puppet@production] dbbackups: Remove all puppet references for db2097, db2098, db2099

https://gerrit.wikimedia.org/r/1040087

Mentioned in SAL (#wikimedia-operations) [2024-06-07T09:43:00Z] <jynus> upgrading and restarting db1239 T360751

Change #1040102 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Upgrade db1239 to MariaDB 10.6

https://gerrit.wikimedia.org/r/1040102

Change #1040102 merged by Jcrespo:

[operations/puppet@production] dbbackups: Upgrade db1239 to MariaDB 10.6

https://gerrit.wikimedia.org/r/1040102

Change #1043802 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Upgrade db1245 to MariaDB 10.6

https://gerrit.wikimedia.org/r/1043802

Mentioned in SAL (#wikimedia-operations) [2024-06-14T14:54:34Z] <jynus> upgrade db1245 to mariadb 10.6 T360751

Change #1043802 merged by Jcrespo:

[operations/puppet@production] dbbackups: Upgrade db1245 to MariaDB 10.6

https://gerrit.wikimedia.org/r/1043802