Sanitarium hosts in codfw (db2186 and db2187) have never been used. They were eventually set up for two reasons
- Set up wikireplicas in codfw
- Failover in case sanitarium hosts in eqiad would fail.
It is pretty clear that #1 will never happen.
In the case of #2, it is a complex process as we'd need to dig into binlogs from each dc and then attempt to move the wikireplicas there. It is a very complex and error prone change and can most likely lead to data corruption, so in the case of breakage of sanitarium hosts, it is more likely that we'd try to fix them or even reclone them from their sanitarium master, rather than risking the whole move to codfw.
Given that, I'd like to remove and repurpose the existing sanitarium hosts in codfw. This would give us two hosts back for production, as currently those hosts do nothing. And also would remove the snowflakes that sanitarium masters are in codfw.
I'd like to hear some thoughts from @taavi @fnegri @Ladsgroup @FCeratto-WMF