@nskaggs @Bstorm If restoring replication turns out to be non-trivial, is it OK to wait until after the failover (in case it turns out it doesn't make sense to try and fix it after all)? In practical terms, the wikis above would be out of sync for over a week.

In T279848#6989283, @LSobanski wrote:

@nskaggs @Bstorm If restoring replication turns out to be non-trivial, is it OK to wait until after the failover (in case it turns out it doesn't make sense to try and fix it after all)? In practical terms, the wikis above would be out of sync for over a week.

If it is not easy to fix replication, we can always try to depool this host entirely and get to serve with labsdb1010 and labsdb1011. Pros: we don't have to worry about labsdb1009. Cons: In the past we've seen labsdb1010 or labsdb1011 struggling (or even crashing) when assuming more load.

Another option if we cannot fix that data drift, would be to put the slave into idempotent mode and assume that its data at least for itwiki on that host, contains drifts from production.

@Marostegui Thanks for jumping on this quickly. Hopefully this means this collection of hosts will be in a healthy state for the migration. To answer your earlier question, I would say yes. If there is another non-trivial breakage let's wait until after switching sanitarium hosts and consider depooling or any other option as well depending on the outcome of the switchover.