Mostly from https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#WebProxy:

1. Stop redis-server and puppet on tools-proxy-03

2. Use hiera to set the active proxy host (toollabs::active_proxy_host) to the hostname (not fqdn) of the newly active proxy. For now this is at https://wikitech.wikimedia.org/wiki/Hiera:Tools. This will update /etc/active-proxy as well as each of the tools-proxy servers for who is the active replication master. There is a period of time where values on individual Tool project instances will not be consistent and this is a race condition possibility for active changes. The safest solution is to the stop redis on the original primary instance to prevent lost changes as the hiera value is updated.

• Update the value in TWO PLACES on wikitech hiera **

3. Run puppet on the DNS recursor hosts (labservices1001 and labservices1002). This is required for internal hits to tools.wmflabs.org to resolve. See https://phabricator.wikimedia.org/diffusion/OPUP/browse/production/modules/role/manifests/labs/dnsrecursor.pp;32dac026174c4eda1383713fda818a6a30c5e981$54

4. Force a puppet run on all the proxies and webgrid nodes
   
   clush -f 10 -g exec-trusty 'sudo puppet agent --test' clush -g webgrid 'sudo puppet agent --test' clush -g all 'sudo puppet agent --test'

5. Ensure puppet is running on tools-proxy-01 and tools-proxy-02 and the replication roles have switched.

5. Test tools.wmflabs.org