When the Redis server should be switched between `tools-redis-1001` and `tools-redis-1002`, https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin#Redis says:
> Redis runs on two instances - tools-redis-01 and -02, and the currently active master is set via hiera on toollabs::active_redis (defaults to tools-redis-01). The other is set to be a slave of the master. Switching over can be done by:
> # Switchover on hiera, set toollabs::active_redis to the hostname (not fqdn) of the up host
> # Force a puppet run on the redis hosts
> # Restart redis on the redis hosts, this resets current connections and makes master / slaves see themselves as master / slave
> # Set the IP address for 'tools-redis.tools.eqiad.wmflabs' and 'tools-redis.eqiad.wmflabs' in hieradata/common/dnsrecursor/labsaliaser.yaml to point to the IP of the new master. This needs a puppet merge + run on the DNS hosts (labservices1001 and holmium as of now). Eventually we'd like to move this step to Horizon...
So there are two truths for "the active Redis server", one in `toollabs::active_redis` (set at https://wikitech.wikimedia.org/wiki/Hiera:Tools; used for `/etc/hosts` and setting up replication between the Redis servers), one in `hieradata/common/dnsrecursor/labsaliaser.yaml` (used for DNS). In addition, the documentation says that the Redis services need to be restarted (I'm not sure that this is actually necessary ATM).
Instead, the truth for "the active Redis server" should only live in `hieradata/common/dnsrecursor/labsaliaser.yaml`. Redis servers should check whether their IP is the same as `tools-redis` and consider themselves master/slave per that. They should automatically restart on changes to the replication direction.
While this requires someone with +2 on `operations/puppet` to switch the Redis server, these changes are rare and often done by administrators with +2.