We have had at least two incidents recently where normal user traffic (i.e. not scraping or denial-of-service) has caused some of our links to get overloaded outbound from Eqiad.
- Our NTT circuit is more used than the other transits at the site
- Due to our confed configuration, and how we set MED=OSPF cost, we have a scenario where an eqord-learnt route can be preferred to one from the same carrier with same as-path length in eqiad
- This will happen if the remote peer is setting a MED on the route and it's higher than the OSPF cost we overwrite the one from eqord with (see T370520).
While we need to review our overall WAN routing policy I think the recent incidents make it clear some action is needed in the short term to balance things a bit better.
To address the first issue we can manually configure certain large eyeball networks to avoid the NTT path in Eqiad, so we get a bit of a better balance. In addition I think we should expand the AS-path on routes learnt in Eqiad from Eqord, in other words make Chicago routes look like they are an additional as-hop away. That will mean Chicago will only be chosen if a path there is better than eqiad.