Enabling Global Transaction ID (https://dev.mysql.com/doc/refman/5.6/en/replication-gtids-concepts.html) should help avoiding having corrupted binary logs and being able to restart the replication process from the exact moment that the secondary stopped, avoiding the drift in the database data caused by skipping binary log entries.
This task is to:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| toolsdb: add gtid_domain_id | operations/puppet | production | +6 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Bstorm | T216208 ToolsDB overload and cleanup | |||
| Resolved | • Bstorm | T216441 Evaluate transferring the non-replicated tables to the new toolsdb server | |||
| Resolved | fnegri | T236101 Find a way to remove non-replicated tables from ToolsDB | |||
| Resolved | None | T306061 Cloud VPS "clouddb-services" project Stretch deprecation | |||
| Resolved | None | T313984 cloudvirt1019: hpssacli not found | |||
| Resolved | fnegri | T301949 ToolsDB upgrade => Bullseye, MariaDB 10.4 | |||
| Resolved | dcaro | T301951 toolsdb: full disk on clouddb1001 broke clouddb1002 (secondary) replication | |||
| Resolved | taavi | T301993 [toolsdb] Enable gtid to help replication recovery |
Change 789588 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] toolsdb: add gtid_domain_id
Note MariaDB and MySQL GTID implementations are different, and the correct documentation is https://mariadb.com/kb/en/gtid/
In theory this would be as simple as
STOP SLAVE; CHANGE MASTER TO MASTER_USE_GTID=slave_pos; START SLAVE;
We don't strictly need gtid_domain_id set before, but I don't see any reason to not set it either, so uploaded a patch to set it like any other production database host. I think we can set that with SET GLOBAL gtid_domain_id without restarting the primary as Puppet changing the config file won't restart the service.
Change 789588 merged by David Caro:
[operations/puppet@production] toolsdb: add gtid_domain_id
This is clouddb1002 (secondary) after setting gtid_domain_id on both servers:
MariaDB [(none)]> SELECT @@GLOBAL.gtid_slave_pos; +------------------------------------------------------+ | @@GLOBAL.gtid_slave_pos | +------------------------------------------------------+ | 0-2886731673-33519859088,2886731673-2886731673-18688 | +------------------------------------------------------+ 1 row in set (0.00 sec)
Is it intentional that there are two entries and the first one starts with 0-?
Mentioned in SAL (#wikimedia-cloud) [2022-05-17T14:12:39Z] <taavi> enable gtid on toolsdb T301993
Looks good:
MariaDB [(none)]> show slave status\G
[...]
Using_Gtid: No
Gtid_IO_Pos:
[...]
MariaDB [(none)]> STOP SLAVE;
Query OK, 0 rows affected (0.01 sec)
MariaDB [(none)]> CHANGE MASTER TO MASTER_USE_GTID=slave_pos;
Query OK, 0 rows affected (0.02 sec)
MariaDB [(none)]> START SLAVE;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> show slave status\G
[...]
Using_Gtid: Slave_Pos
Gtid_IO_Pos: 0-2886731673-33520905397,2886731673-2886731673-13720920
[...]And verified manually that updated rows are getting replicated.