@Dzahn and I did some testing a couple of weeks ago with the intention of unblocking the move of phabricator to codfw so that iridium can be decommissioned.
The plan was to migrate phabricator production from iridium.eqiad.wmnet -> phab2001.codfw.wmnet -> phab1001.eqiad.wmnet. At the end we will have codfw running as a warm backup and we will have documented and validated procedures for migrating and recovering from any serious outage in eqiad.
Unfortunately we almost immediately ran into problems around traffic routing via lvs in codfw.
Phabricator runs two separate ssh services, one is internal for administration and it's connected to the machine's primary ip. The second interface is set up for lvs on git-ssh.wikimedia.org.
Need to investigate further to see which parts of the balancer / lvs misc cluster are missing.
So far, it looks like we need to:
- Add phab2001 here:
- Add codfw here:
- Probably need a temp dns entry for testing git-ssh in codfw? The IP is 10.192.32.149
- phab2001 does not have the lo:lvs interface listening on 10.192.32.149