Well, the biggest issue is that Oracle is deprecating it :-/ https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_format

Re: backup sources, they don't have binlog to start with, so not applicable there.

In T383795#10463084, @jcrespo wrote:

Well, the biggest issue is that Oracle is deprecating it :-/ https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_format

MariaDB isn't doing it

Re: backup sources, they don't have binlog to start with, so not applicable there.

Ah true!

@jcrespo though, now that I think about it, if we want to backup binlogs, I think they should be on RBR anyway, so we have the full status of the rows - but that is to be discussed in some other task.

So my previous comment is my only contribution, I don't feel any way, we would love to have the flexibility of SBR but the reliability of RBR, but we sadly cannot have both. The only thing I can think of is: let's push T207253 and it won't matter!

In T383795#10463134, @Marostegui wrote:

@jcrespo though, now that I think about it, if we want to backup binlogs, I think they should be on RBR anyway, so we have the full status of the rows - but that is to be discussed in some other task.

My intention was to backup both formats, as long as it was possible. I want double backup of everything :-D. But yeah, a discussion for another time, and shouldn't affect this ticket.

In T383795#10463147, @jcrespo wrote:

So my previous comment is my only contribution, I don't feel any way, we would love to have the flexibility of SBR but the reliability of RBR, but we sadly cannot have both. The only thing I can think of is: let's push T207253 and it won't matter!

I talked to @FCeratto-WMF about that one actually today :-)

What we could do is move to SBR, and in parallel try to build the trust and checks in our data and eventually move to RBR (for real this time). Even though I've talked to MariaDB about this to confirm, we never know if in the future they'll stop supporting it.

Thanks for your comments Jaime.
@Ladsgroup @FCeratto-WMF that sounds like a plan?

Change #1111955 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Set RBR to all sanitarium masters

https://gerrit.wikimedia.org/r/1111955

Change #1111955 merged by Marostegui:

[operations/puppet@production] mariadb: Set RBR to all sanitarium masters

https://gerrit.wikimedia.org/r/1111955

I generally like the idea as it opens up the number of potential candidates for master in case master goes down and the designated candidate master doesn't' have the most recent entries.

My worry right now (which is not a major one) would be vendor lock-in. If only MariaDB sticks to keeping SBR and for any reason (redis situation, hostile takeover, etc. etc.) we would need to switch to something else ASAP, it would be quite painful for us.

In T383795#10466314, @Ladsgroup wrote:

I generally like the idea as it opens up the number of potential candidates for master in case master goes down and the designated candidate master doesn't' have the most recent entries.

My worry right now (which is not a major one) would be vendor lock-in. If only MariaDB sticks to keeping SBR and for any reason (redis situation, hostile takeover, etc. etc.) we would need to switch to something else ASAP, it would be quite painful for us.

We are, unfortunately, already locked in due to the way MariaDB implemented replication and GTID

Change #1112040 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Declare RBR in sanitariums.

https://gerrit.wikimedia.org/r/1112040

Change #1112040 merged by Marostegui:

[operations/puppet@production] mariadb: Declare RBR in sanitariums.

https://gerrit.wikimedia.org/r/1112040

I am going to start with s6 for this.
First of all I will put all the hosts with STATEMENT based on hiera, and once everything is migrated, I will move all the defaults in puppet to SBR

Change #1147424 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Move s5 to SBR

https://gerrit.wikimedia.org/r/1147424

In the discussion around T207253 (comparing data across instances) we did not get into a timeline. Perhaps do we want to increase its priority due to the switch to SBR? If so we should also identify how much data is to be compared, how frequently, and what we can do if we detect differences.

In T383795#10833802, @FCeratto-WMF wrote:

In the discussion around T207253 (comparing data across instances) we did not get into a timeline. Perhaps do we want to increase its priority due to the switch to SBR? If so we should also identify how much data is to be compared, how frequently, and what we can do if we detect differences.

I believe @Ladsgroup had a PoC for comparing tables/data but I am not fully sure the state it is, it's been a while.
For basic comparison we could user the user table, as it is relatively fast and it would be rare if it didn't have any additions every 24h.

Change #1147424 merged by Marostegui:

[operations/puppet@production] mariadb: Move s5 to SBR

https://gerrit.wikimedia.org/r/1147424

Mentioned in SAL (#wikimedia-operations) [2025-05-19T10:14:53Z] <marostegui> Move eqiad s5 replicas (except sanitarium master and backup sources) to SBR dbmaint T383795

In T383795#10833857, @Marostegui wrote:

In T383795#10833802, @FCeratto-WMF wrote:

In the discussion around T207253 (comparing data across instances) we did not get into a timeline. Perhaps do we want to increase its priority due to the switch to SBR? If so we should also identify how much data is to be compared, how frequently, and what we can do if we detect differences.

I believe @Ladsgroup had a PoC for comparing tables/data but I am not fully sure the state it is, it's been a while.
For basic comparison we could user the user table, as it is relatively fast and it would be rare if it didn't have any additions every 24h.

it's blocked on making table catalog fully canonical so it can be used.

Mentioned in SAL (#wikimedia-operations) [2025-05-21T07:34:49Z] <marostegui> Move s5 codfw to SBR T383795

Change #1151185 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2*: Remove sanitarium masters

https://gerrit.wikimedia.org/r/1151185

Change #1151185 merged by Marostegui:

[operations/puppet@production] db2*: Remove sanitarium masters

https://gerrit.wikimedia.org/r/1151185

Mentioned in SAL (#wikimedia-operations) [2025-05-28T12:38:55Z] <marostegui> dbmaint x3 codfw make it SBR T383795

Mentioned in SAL (#wikimedia-operations) [2025-05-28T12:53:44Z] <marostegui> dbmaint x3 eqiad make it SBR T383795 T390530

Change #1152070 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2211,db2228: Make them SBR

https://gerrit.wikimedia.org/r/1152070

Change #1152070 merged by Marostegui:

[operations/puppet@production] db2211,db2228: Make them SBR

https://gerrit.wikimedia.org/r/1152070

Change #1153550 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] s6 codfw: Migrate to SBR

https://gerrit.wikimedia.org/r/1153550

Change #1153550 merged by Marostegui:

[operations/puppet@production] s6 codfw: Migrate to SBR

https://gerrit.wikimedia.org/r/1153550

Mentioned in SAL (#wikimedia-operations) [2025-06-04T08:28:21Z] <marostegui> Change s6 codfw dbmaint to SBR T383795

Mentioned in SAL (#wikimedia-operations) [2025-06-04T08:38:05Z] <marostegui> Change s6 eqiad dbmaint to SBR T383795

Change #1153552 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] s6 eqiad: Migrate to SBR

https://gerrit.wikimedia.org/r/1153552

Change #1153552 merged by Marostegui:

[operations/puppet@production] s6 eqiad: Migrate to SBR

https://gerrit.wikimedia.org/r/1153552

Change #1154032 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb s2 codfw: Migrate to SBR

https://gerrit.wikimedia.org/r/1154032

Mentioned in SAL (#wikimedia-operations) [2025-06-05T13:51:31Z] <marostegui> Migrate s2 codfw to SBR dbmaint T383795

Change #1154032 merged by Marostegui:

[operations/puppet@production] mariadb s2 codfw: Migrate to SBR

https://gerrit.wikimedia.org/r/1154032

Change #1154775 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Migrate s2 eqiad to SBR

https://gerrit.wikimedia.org/r/1154775

Mentioned in SAL (#wikimedia-operations) [2025-06-09T09:42:55Z] <marostegui> Migrate s2 eqiad dbmaint to SBR T383795

Change #1154775 merged by Marostegui:

[operations/puppet@production] mariadb: Migrate s2 eqiad to SBR

https://gerrit.wikimedia.org/r/1154775

Change #1166969 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] s3 codfw: Migrate to SBR

https://gerrit.wikimedia.org/r/1166969

Change #1166969 merged by Marostegui:

[operations/puppet@production] s3 codfw: Migrate to SBR

https://gerrit.wikimedia.org/r/1166969

Mentioned in SAL (#wikimedia-operations) [2025-07-08T05:52:43Z] <marostegui> Migrate s3 codfw to SBR T383795

Change #1167142 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] s3 eqiad: Migrate to SBR

https://gerrit.wikimedia.org/r/1167142

Change #1167142 merged by Marostegui:

[operations/puppet@production] s3 eqiad: Migrate to SBR

https://gerrit.wikimedia.org/r/1167142

Mentioned in SAL (#wikimedia-operations) [2025-07-08T07:54:21Z] <marostegui> Migrate s3 eqiad to SBR T383795