@jcrespo I have assigned this to you as the backup owner so we need to coordinate with you, but if during any of the section upgrades you want us to proceed with the backup source upgrade, just let us know.
We'll ask you in any case!

@jcrespo for your information, s2 codfw is almost done upgrading to 10.6, only pending the master.

Thanks for keeping me up to date, will rearrange the backup sources and dbprov accordingly. s6 eqiad and s5 next ?

I believe @ABran-WMF is doing s5 now, but I am not sure.

I am ! T360116 I'll still have to run a replication source switch for eqiad though

@jcrespo s2 fully done. Next is s3.

Change #1018247 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Migrate db2097 backups to db2197

https://gerrit.wikimedia.org/r/1018247

Change #1018276 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Migrate db2098 backups to db2198 and upgrade dbprov2002 to 10.6

https://gerrit.wikimedia.org/r/1018276

In T360751#9699959, @Marostegui wrote:

@jcrespo s2 fully done. Next is s3.

It seems to me that the upgrade pace has increased (good job!). Do you have a guesstimate of how many sections will be in 10.6 more or less fully (even if there is 1 or 2 hosts left) by the end of the quarter, even if aspirationally? That way I can sync by EOQ (my main blocker is how many of the 4 dbprovs to reimage/upgrade to 10.6 by end of June).

I think there will be at least 3 more (s6 is already done) so that would be:
s2 (done), s5 almost done, s3, s7 and s8

As MariaDB 10.4 is EOL in June 18th... we are sort of moving fast

Thanks, that informs me that either 2 or 3 should be upgraded by the end of the quarter, then. Thank you a lot!

Change #1018247 merged by Jcrespo:

[operations/puppet@production] mariadb: Migrate db2097 backups to db2197

https://gerrit.wikimedia.org/r/1018247

@jcrespo eqiad s3 is almost done T362134

Change #1019263 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Move services db2101->db2201,db2099->db2199, upgrade dbprov2003

https://gerrit.wikimedia.org/r/1019263

Change #1019679 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Migrate s5 codfw snapshots to dbprov2005/db2201

https://gerrit.wikimedia.org/r/1019679

Change #1019679 merged by Jcrespo:

[operations/puppet@production] dbbackups: Migrate s5 codfw snapshots to dbprov2005/db2201

https://gerrit.wikimedia.org/r/1019679

Change #1019690 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Upgrade db2139 to MariaDB 10.6

https://gerrit.wikimedia.org/r/1019690

Mentioned in SAL (#wikimedia-operations) [2024-04-15T17:13:53Z] <jynus> stop db2139 dbs for upgrade T360751

Change #1019690 merged by Jcrespo:

[operations/puppet@production] mariadb: Upgrade db2139 to MariaDB 10.6

https://gerrit.wikimedia.org/r/1019690

@Marostegui Update: backups for x1, s2, s6, s5 and s3 are generating dumps and snapshots with MariaDB 10.6 currently on codfw. Doing s5 and s3 on eqiad next. You may see a lot of 10.4 servers, but they are idle and only kept just in case, they are not active, and will be just eventually upgraded or discarded.

Preparing s7 and s8 next.

Thanks, s3 codfw is almost done. Only pending the master which I'll try to do this week

s3 codfw master switched.

@jcrespo I am starting to work with s7 codfw

Change #1020685 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadbd: Upgrade mariadb package on db1240 from 10.4 to 10.6

https://gerrit.wikimedia.org/r/1020685

Change #1020685 merged by Jcrespo:

[operations/puppet@production] mariadbd: Upgrade mariadb package on db1240 from 10.4 to 10.6

https://gerrit.wikimedia.org/r/1020685

Change #1020693 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Upgrade mariadb package on db1216 from 10.4 to 10.6

https://gerrit.wikimedia.org/r/1020693

Change #1020693 merged by Jcrespo:

[operations/puppet@production] mariadb: Upgrade mariadb package on db1216 from 10.4 to 10.6

https://gerrit.wikimedia.org/r/1020693

Change #1020697 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] site.pp: Reorder backup sources by server name and update comments

https://gerrit.wikimedia.org/r/1020697

Change #1020697 merged by Jcrespo:

[operations/puppet@production] site.pp: Reorder backup sources by server name and update comments

https://gerrit.wikimedia.org/r/1020697

update: on both eqiad and codfw we are generating dumps and snapshots in 10.6 for x1, s2, s6, s5, s3.

I am seeing that s7 and s4 are next. Are you aiming to complete all sections by the end of the quarter (not caring about the top primaries)?

yes we're aiming to that!

Change #1025710 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] mariadb: Upgrade db1150, db1171 and move s4, s7, s8 backups to 10.6

https://gerrit.wikimedia.org/r/1025710

Change #1018276 abandoned by Jcrespo:

[operations/puppet@production] mariadb: Migrate db2098 backups to db2198 and upgrade dbprov2002 to 10.6

Reason:

in favour of a more up to date change

https://gerrit.wikimedia.org/r/1018276

Change #1019263 abandoned by Jcrespo:

[operations/puppet@production] mariadb: Move services db2101->db2201,db2099->db2199, upgrade dbprov2003

Reason:

in favour of a more up to date change

https://gerrit.wikimedia.org/r/1019263