Currently eqiad and codfw are sharing all their external routing:
- inbound: both codfw and eqiad IP prefixes (one /23 each) are advertised from both sites (and eqord) as a single /22
- outbound: both sites exchange their full views
Which mean that traffic toward codfw VIPs can enter our network via eqiad (and the other way around).
And traffic exiting a codfw can be router to eqiad before an external transit/peering (and the other way around).
My guess is that is has been configured like that in the past so bandwidth from both sites was being utilized.
However this might not be the best option anymore, the main issue being with traffic engineering.
When esams is depooled, eqiad links tend to saturate. For that reason the traffic team used a different DNS geo-maps file to redirect some US regions to codfw instead of eqiad. But with the current routing strategy, even if the traffic goes to codfw's VIPs, it can still enters through eqiad, potentially saturating links.
The other reason to change this, is so both sites are more independent, splitting the /22 into two /23s would make a miss-configuration less likely to impact both sites. Similarly, not exchanging full views would make routing simpler (similar to our other sites) and convergence faster.