The current Puppettization of the semi-sync replication for the Mariadb module is coupled with the value of $::mw_primary.
There is no need anymore to keep them coupled and semi-sync can be enabled independently of it.
Description
Description
Details
Details
| Project | Branch | Lines +/- | Subject | |
|---|---|---|---|---|
| operations/puppet | production | +7 -12 | mariadb-core: Decouple Mariadb semi-sync replication from $::mw_primary |
Event Timeline
Comment Actions
Change 344442 had a related patch set uploaded (by Jcrespo):
[operations/puppet@production] mariadb-core: Decouple Mariadb semi-sync replication from $::mw_primary
Comment Actions
@jcrespo if I understand the patch correctly this means that we'll activate semi-sync also for the cross-DC replication?
If so I would consider not having it for the cross-DC or gather some data to ensure that the threshold is not usually reach to avoid it flapping between async and semi-sync.
See also the data I've gathered ~1 year ago in T131753
Comment Actions
Actually, I am not sure that is needed- things flopped because ALL servers took more than 10 ms to apply the changes, that doesn't make the master flop if 1 server takes, let's say, 200 ms to acknowledge the issue. If you can convince me that that can cause problems (I will set it up manually on a shard) or how my logic is flawed, I will change it to master-only.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-03-27T08:42:16Z] <jynus> deploying semisync replication to all hosts (eqiad and codfw) on s6 T161007
Comment Actions
I think this will work with no issue and no impact on production, while mirroring full the setup on both dcs:
root@db2028[(none)]> SHOW GLOBAL VARIABLES like 'Rpl%'; +------------------------------------+-------+ | Variable_name | Value | +------------------------------------+-------+ | rpl_recovery_rank | 0 | | rpl_semi_sync_master_enabled | ON | | rpl_semi_sync_master_timeout | 100 | | rpl_semi_sync_master_trace_level | 32 | | rpl_semi_sync_master_wait_no_slave | ON | | rpl_semi_sync_slave_enabled | ON | | rpl_semi_sync_slave_trace_level | 32 | +------------------------------------+-------+ 7 rows in set (0.00 sec) root@db2028[(none)]> SHOW GLOBAL STATUS like 'Rpl%'; +--------------------------------------------+-------------+ | Variable_name | Value | +--------------------------------------------+-------------+ | Rpl_semi_sync_master_clients | 6 | | Rpl_semi_sync_master_net_avg_wait_time | 693 | | Rpl_semi_sync_master_net_wait_time | 70530904 | | Rpl_semi_sync_master_net_waits | 101670 | | Rpl_semi_sync_master_no_times | 1 | | Rpl_semi_sync_master_no_tx | 32000 | | Rpl_semi_sync_master_status | ON | | Rpl_semi_sync_master_timefunc_failures | 0 | | Rpl_semi_sync_master_tx_avg_wait_time | 1003 | | Rpl_semi_sync_master_tx_wait_time | 42604411 | | Rpl_semi_sync_master_tx_waits | 42438 | | Rpl_semi_sync_master_wait_pos_backtraverse | 0 | | Rpl_semi_sync_master_wait_sessions | 0 | | Rpl_semi_sync_master_yes_tx | 42372 | | Rpl_semi_sync_slave_status | ON | | Rpl_status | AUTH_MASTER | +--------------------------------------------+-------------+ 16 rows in set (0.00 sec)
Comment Actions
I'm not saying it will not work, I just suggested to monitor it, because the ideal rpl_semi_sync_master_timeout that minimize the the waiting while keeping almost always the semi-sync active might be different between the slaves in the same DC and the cross-DC replication due to higher latency.
Comment Actions
If so I would consider not having it for the cross-DC or gather some data to ensure that the threshold is not usually reach to avoid it flapping between async and semi-sync.
the ideal rpl_semi_sync_master_timeout that minimize the the waiting while keeping almost always the semi-sync active might be different between the slaves in the same DC and the cross-DC replication due to higher latency.
From the comment, I think you are getting confused about how semisync works on mysql. Unless all slaves fail at the same time from a single datacenter (in which case we will have worse problems), semisync enabled over the DC will have absolutely no consequence- master only waits for the first n slaves, where n (rpl_semi_sync_master_wait_for_slave_count) is equal to 1 for MariaDB 10. In a nutshell, semisyncronous is a master-only thing, and adding extra slaves (no matter how slow they are, has no slowdown on the master.
What you may be thinking is galera-style timeouts/cassandra ALL quorums, where the "master" or transaction coordinator waits for *all* slaves to ack the transaction, and over the dc it will not work at all, slowing the replication down to failure.
Comment Actions
Right, I was forgetting the details of the implementation, I agree that it might affect only the misc shards where we just have one slave in the same DC. No objections.
Comment Actions
Change 344442 merged by Jcrespo:
[operations/puppet@production] mariadb-core: Decouple Mariadb semi-sync replication from $::mw_primary
Comment Actions
This is now deployed, only applying to shards other than s6 is left.
The other usage of $::mw_primary (do not page for the non-primary dc) can be left being fully asynchronous.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2017-03-27T12:44:51Z] <jynus> deploying semi-sync replication to all hosts on codfw T161007
Comment Actions
Semi sync is deployed on all masters, independently of the dc:
root@neodymium:~$ cumin 'R:Class = Role::Mariadb::Groups and R:Class%mysql_group = core and R:Class%mysql_role = master' "mysql --skip-ssl -e \"SHOW GLOBAL STATUS like 'Rpl_semi_sync_master_clients'\"" 20 hosts will be targeted: db[2016-2019,2023,2028-2029,2033].codfw.wmnet,db[1018,1031,1040-1041,1049-1050,1052,1075].eqiad.wmnet,es[2016,2018].codfw.wmnet,es[1011,1014].eqiad.wmnet Confirm to continue [y/n]? y ===== NODE GROUP ===== (1) db1052.eqiad.wmnet ----- OUTPUT of 'mysql --skip-ssl...master_clients'"' ----- Variable_name Value Rpl_semi_sync_master_clients 11 ===== NODE GROUP ===== (1) db1018.eqiad.wmnet ----- OUTPUT of 'mysql --skip-ssl...master_clients'"' ----- Variable_name Value Rpl_semi_sync_master_clients 10 ===== NODE GROUP ===== (4) es[2016,2018].codfw.wmnet,es[1011,1014].eqiad.wmnet ----- OUTPUT of 'mysql --skip-ssl...master_clients'"' ----- Variable_name Value Rpl_semi_sync_master_clients 3 ===== NODE GROUP ===== (1) db1031.eqiad.wmnet ----- OUTPUT of 'mysql --skip-ssl...master_clients'"' ----- Variable_name Value Rpl_semi_sync_master_clients 2 ===== NODE GROUP ===== (1) db2033.codfw.wmnet ----- OUTPUT of 'mysql --skip-ssl...master_clients'"' ----- Variable_name Value Rpl_semi_sync_master_clients 1 ===== NODE GROUP ===== (3) db[2016-2017].codfw.wmnet,db1075.eqiad.wmnet ----- OUTPUT of 'mysql --skip-ssl...master_clients'"' ----- Variable_name Value Rpl_semi_sync_master_clients 7 ===== NODE GROUP ===== (4) db[2019,2023,2028-2029].codfw.wmnet ----- OUTPUT of 'mysql --skip-ssl...master_clients'"' ----- Variable_name Value Rpl_semi_sync_master_clients 6 ===== NODE GROUP ===== (1) db2018.codfw.wmnet ----- OUTPUT of 'mysql --skip-ssl...master_clients'"' ----- Variable_name Value Rpl_semi_sync_master_clients 5 ===== NODE GROUP ===== (3) db[1040-1041,1050].eqiad.wmnet ----- OUTPUT of 'mysql --skip-ssl...master_clients'"' ----- Variable_name Value Rpl_semi_sync_master_clients 9 ===== NODE GROUP ===== (1) db1049.eqiad.wmnet ----- OUTPUT of 'mysql --skip-ssl...master_clients'"' ----- Variable_name Value Rpl_semi_sync_master_clients 8 ================ PASS |████████████████████████████████████████████████████████████████████████████████████████████| 100% (20/20) [00:00<00:00, 28.19hosts/s] FAIL | | 0% (0/20) [00:00<?, ?hosts/s] 100.0% (20/20) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...master_clients'"'. 100.0% (20/20) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
We could consider add a slave for x1 on codfw (db2033), but otherwise this is complete. CC @Marostegui
Comment Actions
+1 to this, to cover our back and sleep better.
We can discuss that on a separate ticket, but I think we should do it if we have HW available.