Going to mark x2 as done as we don't use GTID there (we don't care about data consistency on that section)

I am running into some issues while cleaning up this on m1. Some of the domain_ids cannot be cleaned up and they error out with:

root@db1117.eqiad.wmnet[(none)]> flush binary logs delete_domain_id=(171966484);
ERROR 1076 (HY000): Could not delete gtid domain. Reason: binlog files may contain gtids from the domain ('171966484') being deleted. Make sure to first purge those files.

That domain_id belong to es1027 which it's never been in m1, and I guess it is just an IP that was re-used. Anyhow, that gtid_domain_id isn't on any of the binlogs. So I am investigating as we might find this issue on some other sections.

This is interesting....it looks like flush binary logs does check local files and that domain_id is part of these files:

-rw-rw---- 1 mysql mysql 1004M Mar 14  2018 db1016-bin.013077
-rw-rw---- 1 mysql mysql 1001M Mar 14  2018 db1016-bin.013078
-rw-rw---- 1 mysql mysql 1001M Mar 14  2018 db1016-bin.013079
-rw-rw---- 1 mysql mysql 1001M Mar 14  2018 db1016-bin.013080

They are old leftover files of course.

root@db1117:/srv/sqldata.m1# ls | grep "db1016-bin" | wc -l
78

Purging those made no difference. There are no more binary logs on db1117 or db1195 (master) which contain that gtid (at least on file). So I am checking where that error is coming from.

I have ended up filing a MariaDB bug: https://jira.mariadb.org/browse/MDEV-30386

My last tests have been oriented to use RESET MASTER, which is the approach that looks less error prone, although it is the more invasive one (as it requires reconfiguring all the replicas).
The tests I have done, so far would require:

• Read only (MW and mysql)
• Reset master on the master itself, this unconfigures ALL the direct replicas.
  • We'd need to check what would be the status with the intermediate master, probably the safest thing would be to do it first on codfw and then keep going up (wiki replicas).
    • Although we need to check what happens with intermediate masters replicating gtid_domain_id downstream.
• Issue a truncate truncate gtid_slave_pos; on all the involved hosts.
• Issue a stop slave; SET GLOBAL gtid_slave_pos = ''; on the slaves
• Reconfigure all the slaves
  • Start replication and then issue STOP SLAVE; CHANGE MASTER TO MASTER_USE_GTID=Slave_pos; START SLAVE;

This is the initial draft, but I really need to set up a testing environment where I have an intermediate master so I can test the intermediate master gtid_domain_id replication when the first reset master is done on the intermediate master.