Page MenuHomePhabricator

Decouple Mariadb semi-sync replication from $::mw_primary
Closed, ResolvedPublic

Description

The current Puppettization of the semi-sync replication for the Mariadb module is coupled with the value of $::mw_primary.
There is no need anymore to keep them coupled and semi-sync can be enabled independently of it.

Details

Related Gerrit Patches:

Event Timeline

Volans created this task.Mar 21 2017, 4:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 21 2017, 4:38 PM
jcrespo moved this task from Triage to Next on the DBA board.Mar 21 2017, 4:47 PM

This should not take much time, setting up next on the pipeline.

jcrespo moved this task from Next to In progress on the DBA board.Mar 23 2017, 12:22 PM
jcrespo claimed this task.Mar 23 2017, 12:41 PM

Change 344442 had a related patch set uploaded (by Jcrespo):
[operations/puppet@production] mariadb-core: Decouple Mariadb semi-sync replication from $::mw_primary

https://gerrit.wikimedia.org/r/344442

@jcrespo if I understand the patch correctly this means that we'll activate semi-sync also for the cross-DC replication?
If so I would consider not having it for the cross-DC or gather some data to ensure that the threshold is not usually reach to avoid it flapping between async and semi-sync.
See also the data I've gathered ~1 year ago in T131753

So s/both/master/ ?

Actually, I am not sure that is needed- things flopped because ALL servers took more than 10 ms to apply the changes, that doesn't make the master flop if 1 server takes, let's say, 200 ms to acknowledge the issue. If you can convince me that that can cause problems (I will set it up manually on a shard) or how my logic is flawed, I will change it to master-only.

I will set this up on s6-codfw as a test, but next week.

Mentioned in SAL (#wikimedia-operations) [2017-03-27T08:42:16Z] <jynus> deploying semisync replication to all hosts (eqiad and codfw) on s6 T161007

I think this will work with no issue and no impact on production, while mirroring full the setup on both dcs:

root@db2028[(none)]> SHOW GLOBAL VARIABLES like 'Rpl%';
+------------------------------------+-------+
| Variable_name                      | Value |
+------------------------------------+-------+
| rpl_recovery_rank                  | 0     |
| rpl_semi_sync_master_enabled       | ON    |
| rpl_semi_sync_master_timeout       | 100   |
| rpl_semi_sync_master_trace_level   | 32    |
| rpl_semi_sync_master_wait_no_slave | ON    |
| rpl_semi_sync_slave_enabled        | ON    |
| rpl_semi_sync_slave_trace_level    | 32    |
+------------------------------------+-------+
7 rows in set (0.00 sec)

root@db2028[(none)]> SHOW GLOBAL STATUS like 'Rpl%';
+--------------------------------------------+-------------+
| Variable_name                              | Value       |
+--------------------------------------------+-------------+
| Rpl_semi_sync_master_clients               | 6           |
| Rpl_semi_sync_master_net_avg_wait_time     | 693         |
| Rpl_semi_sync_master_net_wait_time         | 70530904    |
| Rpl_semi_sync_master_net_waits             | 101670      |
| Rpl_semi_sync_master_no_times              | 1           |
| Rpl_semi_sync_master_no_tx                 | 32000       |
| Rpl_semi_sync_master_status                | ON          |
| Rpl_semi_sync_master_timefunc_failures     | 0           |
| Rpl_semi_sync_master_tx_avg_wait_time      | 1003        |
| Rpl_semi_sync_master_tx_wait_time          | 42604411    |
| Rpl_semi_sync_master_tx_waits              | 42438       |
| Rpl_semi_sync_master_wait_pos_backtraverse | 0           |
| Rpl_semi_sync_master_wait_sessions         | 0           |
| Rpl_semi_sync_master_yes_tx                | 42372       |
| Rpl_semi_sync_slave_status                 | ON          |
| Rpl_status                                 | AUTH_MASTER |
+--------------------------------------------+-------------+
16 rows in set (0.00 sec)

I'm not saying it will not work, I just suggested to monitor it, because the ideal rpl_semi_sync_master_timeout that minimize the the waiting while keeping almost always the semi-sync active might be different between the slaves in the same DC and the cross-DC replication due to higher latency.

If so I would consider not having it for the cross-DC or gather some data to ensure that the threshold is not usually reach to avoid it flapping between async and semi-sync.

the ideal rpl_semi_sync_master_timeout that minimize the the waiting while keeping almost always the semi-sync active might be different between the slaves in the same DC and the cross-DC replication due to higher latency.

From the comment, I think you are getting confused about how semisync works on mysql. Unless all slaves fail at the same time from a single datacenter (in which case we will have worse problems), semisync enabled over the DC will have absolutely no consequence- master only waits for the first n slaves, where n (rpl_semi_sync_master_wait_for_slave_count) is equal to 1 for MariaDB 10. In a nutshell, semisyncronous is a master-only thing, and adding extra slaves (no matter how slow they are, has no slowdown on the master.

What you may be thinking is galera-style timeouts/cassandra ALL quorums, where the "master" or transaction coordinator waits for *all* slaves to ack the transaction, and over the dc it will not work at all, slowing the replication down to failure.

Right, I was forgetting the details of the implementation, I agree that it might affect only the misc shards where we just have one slave in the same DC. No objections.

Change 344442 merged by Jcrespo:
[operations/puppet@production] mariadb-core: Decouple Mariadb semi-sync replication from $::mw_primary

https://gerrit.wikimedia.org/r/344442

This is now deployed, only applying to shards other than s6 is left.

The other usage of $::mw_primary (do not page for the non-primary dc) can be left being fully asynchronous.

Mentioned in SAL (#wikimedia-operations) [2017-03-27T12:44:51Z] <jynus> deploying semi-sync replication to all hosts on codfw T161007

jcrespo closed this task as Resolved.Mar 27 2017, 3:47 PM
jcrespo added a subscriber: Marostegui.

Semi sync is deployed on all masters, independently of the dc:

root@neodymium:~$ cumin 'R:Class = Role::Mariadb::Groups and R:Class%mysql_group = core and R:Class%mysql_role = master' "mysql --skip-ssl -e \"SHOW GLOBAL STATUS like 'Rpl_semi_sync_master_clients'\""
20 hosts will be targeted:
db[2016-2019,2023,2028-2029,2033].codfw.wmnet,db[1018,1031,1040-1041,1049-1050,1052,1075].eqiad.wmnet,es[2016,2018].codfw.wmnet,es[1011,1014].eqiad.wmnet
Confirm to continue [y/n]? y
===== NODE GROUP =====                                                                                                                               
(1) db1052.eqiad.wmnet                                                                                                                               
----- OUTPUT of 'mysql --skip-ssl...master_clients'"' -----                                                                                          
Variable_name   Value                                                                                                                                
Rpl_semi_sync_master_clients    11                                                                                                                   
===== NODE GROUP =====                                                                                                                               
(1) db1018.eqiad.wmnet                                                                                                                               
----- OUTPUT of 'mysql --skip-ssl...master_clients'"' -----                                                                                          
Variable_name   Value                                                                                                                                
Rpl_semi_sync_master_clients    10                                                                                                                   
===== NODE GROUP =====                                                                                                                               
(4) es[2016,2018].codfw.wmnet,es[1011,1014].eqiad.wmnet                                                                                              
----- OUTPUT of 'mysql --skip-ssl...master_clients'"' -----                                                                                          
Variable_name   Value                                                                                                                                
Rpl_semi_sync_master_clients    3                                                                                                                    
===== NODE GROUP =====                                                                                                                               
(1) db1031.eqiad.wmnet                                                                                                                               
----- OUTPUT of 'mysql --skip-ssl...master_clients'"' -----                                                                                          
Variable_name   Value                                                                                                                                
Rpl_semi_sync_master_clients    2                                                                                                                    
===== NODE GROUP =====                                                                                                                               
(1) db2033.codfw.wmnet                                                                                                                               
----- OUTPUT of 'mysql --skip-ssl...master_clients'"' -----                                                                                          
Variable_name   Value                                                                                                                                
Rpl_semi_sync_master_clients    1                                                                                                                    
===== NODE GROUP =====                                                                                                                               
(3) db[2016-2017].codfw.wmnet,db1075.eqiad.wmnet                                                                                                     
----- OUTPUT of 'mysql --skip-ssl...master_clients'"' -----                                                                                          
Variable_name   Value                                                                                                                                
Rpl_semi_sync_master_clients    7                                                                                                                    
===== NODE GROUP =====                                                                                                                               
(4) db[2019,2023,2028-2029].codfw.wmnet                                                                                                              
----- OUTPUT of 'mysql --skip-ssl...master_clients'"' -----                                                                                          
Variable_name   Value                                                                                                                                
Rpl_semi_sync_master_clients    6                                                                                                                    
===== NODE GROUP =====                                                                                                                               
(1) db2018.codfw.wmnet                                                                                                                               
----- OUTPUT of 'mysql --skip-ssl...master_clients'"' -----                                                                                          
Variable_name   Value                                                                                                                                
Rpl_semi_sync_master_clients    5                                                                                                                    
===== NODE GROUP =====                                                                                                                               
(3) db[1040-1041,1050].eqiad.wmnet                                                                                                                   
----- OUTPUT of 'mysql --skip-ssl...master_clients'"' -----                                                                                          
Variable_name   Value                                                                                                                                
Rpl_semi_sync_master_clients    9                                                                                                                    
===== NODE GROUP =====                                                                                                                               
(1) db1049.eqiad.wmnet                                                                                                                               
----- OUTPUT of 'mysql --skip-ssl...master_clients'"' -----                                                                                          
Variable_name   Value                                                                                                                                
Rpl_semi_sync_master_clients    8                                                                                                                    
================                                                                                                                                     
PASS |████████████████████████████████████████████████████████████████████████████████████████████| 100% (20/20) [00:00<00:00, 28.19hosts/s]         
FAIL |                                                                                                     |   0% (0/20) [00:00<?, ?hosts/s]         
100.0% (20/20) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...master_clients'"'.
100.0% (20/20) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.

We could consider add a slave for x1 on codfw (db2033), but otherwise this is complete. CC @Marostegui

We could consider add a slave for x1 on codfw (db2033), but otherwise this is complete. CC @Marostegui

+1 to this, to cover our back and sleep better.
We can discuss that on a separate ticket, but I think we should do it if we have HW available.