Currently this project is puppetized on wikitech via https://wikitech.wikimedia.org/wiki/Hiera:Project-proxy -- I'm going to fix that /after/ this fail-over is done.
The API service mentioned below is a uwsgi service called 'invisible_unicorn'.
These steps will not result in downtime:
[] Create new eqiad1 proxy nodes, proxy-01 and proxy-02
[] Add proxy-01 and proxy-02 to $all_proxies, let puppet update
[] ensure that redis is syncing properly between regions
These steps will result in partial downtime with creating/deleting proxies:
[] Set $active_proxy to point to proxy-01, let puppet update
[] stop puppet and the API on novaproxy-01
[] stop api on proxy-01, restore database (it's on NFS, available to all nodes), restart API there
[] Update Horizon and others to reference the api on proxy-01
[] Test! Everything should be working now, although novaproxy-01 is proxying and proxy-01 is hosting the API
These step should not result in any downtime:
[] Update proxy DNS records (via script, to be determined)
[] test some more