Recapitulating what I *understand* to be the best thing to do:

Prep work that still needs to be completed:

1. Reimage to jessie the four eqiad redis masters (this should be discussed first)
2. Add two new servers to the codfw cluster and set up encryption and replication between the datacenters
3. Upgrade mediawiki-config to point to the local resources in each datacenter
4. Ensure that the jobrunners in the non-active datacenters are not started

The switchover should follow this procedure:

1. Jobrunners in eqiad get stopped
2. mediawiki goes read-only - this should ensure no new job gets enqueued, right?
3. mediawiki primary gets switched in mediawiki-config and in puppet
   1. As a consequence the redis replication will be inverted
4. mediawiki is set read-write in codfw
5. We start the jobrunners in codfw and they will consume the jobs left over from eqiad

While complex and requiring a series of manual steps, this should be manageable to do. I'd like others to comment on this plan: the main uncertainty for me is how are shards caluculated: if in any place we use hostname/port instead of a label for sharding, it might be a problem.

Apart from that, we should be sure that jobs will be correctly executed when restarted.

mediawiki goes read-only - this should ensure no new job gets enqueued, right?

That is what I would expect, but I could not verify last time I did it (however, I could not distinguish traffic from new jobs vs. already enqueued at the time), as I did not stop the queues at that time.

See also T128730 for comments on doing single-DC maintenance.

In T124673#2099319, @Joe wrote:

While complex and requiring a series of manual steps, this should be manageable to do. I'd like others to comment on this plan: the main uncertainty for me is how are shards caluculated: if in any place we use hostname/port instead of a label for sharding, it might be a problem.

Apart from that, we should be sure that jobs will be correctly executed when restarted.

The steps above make sense. As for sharding, the 'rdb*' tag names in config are meant to have servers with the same data in each DC for simplicity. So rdb1 in eqiad would have the same data as rdb1 in codfw, whatever the actual host names are.

Strictly speaking, even the tag names don't have to match as long all 4 eqiad servers have codfw "masters" with the same data. So you could have rdb0-3 in one DC and rdb1-4 in another, though, that would be needlessly confusing.

All the blockers I listed earlier have been removed. Resolving and moving documentation to https://wikitech.wikimedia.org/wiki/Switch_Datacenter#Job_queue

The procedure is simpler than expected as redis can detect circular replications and will just refuse to communicate to its slave, if such slave is also a master. So we should be safe during the transition even if the puppet runs are not all in sync.