Looking at some stats I see we barely use our Lumen transit in Codfw.

The reason is that most prefixes we see with the same AS-path length from Arelion and Lumen. But Arelion have MED=0 on the routes they send, and Lumen have various values higher than that, which is getting used as a tie-break, for example:

cmooney@re0.cr2-codfw> show route protocol bgp 176.61.33.2 terse 

inet.0: 937054 destinations, 3246024 routes (936878 active, 61 holddown, 419 hidden)
Restart Complete
+ = Active Route, - = Last Active, * = Both

A V Destination        P Prf   Metric 1   Metric 2  Next hop        AS path
* V 176.61.0.0/18      B 170        100          0                  1299 6830 ?
  valid                                            >208.80.153.218
                                                    208.80.153.213
  V                    B 170        100      22640                  3356 6830 ?
  valid                                            >64.156.73.169

We also had the recent complications from accepting MED inbound from Amazon in eqord and eqiad and how we use it internally.

We can wait for the larger evaluation (T297355), but I thought it might be worth discussing either way as it's part of the equation. For my part I don't believe it is serving us well to tie-break on this attribute. My logic being:

• We should ultimately control the policy about where we send outbound traffic
• Different carriers have different policies, like this between Lumen and Arelion
  • Whatever Lumen may be trying to signal, it's definitely not "use Arelion instead"
• We should be doing hot-potato routing from our sites towards the internet
  • So the only valid MED comparison would be between two routes from the same carrier at the same site

I think it would be reasonable, and give us more control plus make things easier to reason about, to just set MED=0 on all inbound routes so it is not used as a tie-break. Either in BGP_sanitize_in, or possibly if we didn't want to inlude peers in BGP_transit_in.