failing over graphite in T85909 was not fun nor fast, namely doing it via dns has several disavantages and mostly because clients are long-running and won't pick up dns changes by themselves.
* diamond needs restart (doesn't use dns but needs a fleetwide restart anyway)
* jmxtrans needs restart (used to introspect kafka mostly)
* txstatsd needs restart (deployed locally on swift boxes and varnishkafka (thus cache boxes) ATM)
* reqstats/udp2log lives on analytics boxes, thus firewall needs to be changed to allow traffic
the transition can be somewhat smoothed by redirecting carbon traffic from the original host to the new one (changing txstatsd and carbon-relay to forward to the new machine instead of localhost) but still a painful and long process, especially the restarts