Change Details

We use both GTID and pt-heartbeat, which is problematic if we want to do cross-DC DB requests (since both mechanisms require actions from the master DB process). For more context cf. T172497 and in particular T172497#4905268 .

We use both GTID and pt-heartbeat, which is problematic if we want to do cross-DC DB requests (since both mechanisms require actions from the master DB process). For more context cf. T172497 and in particular T172497#4905268 . **Jaime's summary:** I am not saying GTID should be removed, but we should have a discussion to understand it and either fix it or consider a replacement. More details about issues: * GTID has many practical problems ** MySQL and MariaDB had different implementations, which is a big deal because it means we cannot transition from one to the other. While this is not a blocker for Cross-dc itself, if alternatives are searched, this should be taken into account ** GTID is causing operational issues- in theory it allows easy handling of databases, but in reality it is a mess we actually stopped using operationally because how many bugs it has ** GTID is causing mediawiki issues- every time a master failover happens (what GTID promises it does nicely), jobs fail unable to locate gtid position because it varies from master to master. This causues complains by other developers and production errors. ** GTID string keeps growing and growing coordinates, in a useless way, while only a master (an id) is actually writing at a time * Chronology protector doesn't work well on master switch. A master change (be it within dc or cross-dc) should be taken into account in the logic, (probably due to the above issues). Ideally, also, a replication checker would avoid querying the master * Replication control has issues cross-dc- spikes of lag are very likely between datacenters, either due to the added latency, the limited bandwidth or the extra unreliability. This was not an issue until now because the passive database could just get delayed. But what will it happen when we have reads on the non-primary dc? Should we stop writes until the secondary datacenter catches up? Should we consider the secondary dc as "down". Should we be permissive against delayed reads. Should we switch datacenter? All options have problems- stopping writes (and reads), may impact not only the throughput of the replicated dc, but the primary dc too (many batched actions happens every week that would be 10x or slower. Can the secondary dc (or some queries) accept more delay? How should mediawiki behave in case of a split brain (network lost to secondary dc), should it wait forever, or is it a timeout ok- could that impact the number of open connections to the primary master? * pt-heartbeat has been working nicely for what we need (it may need changes T172497, but that is on the DBAs), however it cannot be at the moment a proper substitute for GTID because it is not "continuous", it just generate events every second (currently), so it is not a proper substitute. Should frequency be changed or a new event be generated within the transaction each time chronology wants to be changed to be a viable alternative? Is pt-hearbeat being used transactionally consistent by Mediawiki (as far as I know, it isn't, it tries to use real-world clock rather than "database time", which is certainly leading to problems). * What are proper logical, architecture and operational solutions to the above issues?

We use both GTID and pt-heartbeat, which is problematic if we want to do cross-DC DB requests (since both mechanisms require actions from the master DB process). For more context cf. T172497 and in particular T172497#4905268 . **Jaime's summary:** I am not saying GTID should be removed, but we should have a discussion to understand it and either fix it or consider a replacement. More details about issues: * GTID has many practical problems ** MySQL and MariaDB had different implementations, which is a big deal because it means we cannot transition from one to the other. While this is not a blocker for Cross-dc itself, if alternatives are searched, this should be taken into account ** GTID is causing operational issues- in theory it allows easy handling of databases, but in reality it is a mess we actually stopped using operationally because how many bugs it has ** GTID is causing mediawiki issues- every time a master failover happens (what GTID promises it does nicely), jobs fail unable to locate gtid position because it varies from master to master. This causues complains by other developers and production errors. ** GTID string keeps growing and growing coordinates, in a useless way, while only a master (an id) is actually writing at a time * Chronology protector doesn't work well on master switch. A master change (be it within dc or cross-dc) should be taken into account in the logic, (probably due to the above issues). Ideally, also, a replication checker would avoid querying the master * Replication control has issues cross-dc- spikes of lag are very likely between datacenters, either due to the added latency, the limited bandwidth or the extra unreliability. This was not an issue until now because the passive database could just get delayed. But what will it happen when we have reads on the non-primary dc? Should we stop writes until the secondary datacenter catches up? Should we consider the secondary dc as "down". Should we be permissive against delayed reads. Should we switch datacenter? All options have problems- stopping writes (and reads), may impact not only the throughput of the replicated dc, but the primary dc too (many batched actions happens every week that would be 10x or slower. Can the secondary dc (or some queries) accept more delay? How should mediawiki behave in case of a split brain (network lost to secondary dc), should it wait forever, or is it a timeout ok- could that impact the number of open connections to the primary master? * pt-heartbeat has been working nicely for what we need (it may need changes T172497, but that is on the DBAs), however it cannot be at the moment a proper substitute for GTID because it is not "continuous", it just generate events every second (currently), so it is not a proper substitute. Should frequency be changed or a new event be generated within the transaction each time chronology wants to be changed to be a viable alternative? Is pt-hearbeat being used transactionally consistent by Mediawiki (as far as I know, it isn't, it tries to use real-world clock rather than "database time", which is certainly leading to problems). * What are proper logical, architecture and operational solutions to the above issues?