Change Details

Looking at some stats I see we barely use our Lumen transit in Codfw. The reason is that most prefixes we see with the same AS-path length from Arelion and Lumen. But Arelion have MED=0 on the routes they send, and Lumen have various values higher than that, which is getting used as a tie-break, for example: ``` cmooney@re0.cr2-codfw> show route protocol bgp 176.61.34.2 terse inet.0: 937054 destinations, 3246024 routes (936878 active, 61 holddown, 419 hidden) Restart Complete + = Active Route, - = Last Active, * = Both A V Destination P Prf Metric 1 Metric 2 Next hop AS path * V 176.61.0.0/18 B 170 100 0 1299 6830 ? valid >208.80.153.218 208.80.153.213 V B 170 100 22640 3356 6830 ? valid >64.156.73.169 ``` We also had the recent complications from accepting MED inbound from Amazon in eqord and eqiad and how we use it internally. We can wait here for the larger evaluation, but I thought it might be worth discussing here either way as it's part of the equation. From my part I don't believe it is serving us well to tie-break on this attribute. My logic being: * We should ultimately control the policy about where we send outbound traffic * Different carriers have different policies, like this between Lumen and Arelion ** Whatever Lumen may be trying to signal, it's definitely not "use Arelion instead" * We should be doing hot-potato routing from our sites towards the internet ** So the only valid MED comparison would be between two routes from the same carrier at the same site I think it would be reasonable, and give us more control plus make things easier to reason about, to just set MED=0 on all inbound routes so it is not used as a tie-break. Either in //BGP_sanitize_in//, or possibly if we didn't want to inlude peers in //BGP_transit_in//.

Looking at some stats I see we barely use our Lumen transit in Codfw. The reason is that most prefixes we see with the same AS-path length from Arelion and Lumen. But Arelion have MED=0 on the routes they send, and Lumen have various values higher than that, which is getting used as a tie-break, for example: ``` cmooney@re0.cr2-codfw> show route protocol bgp 176.61.33.2 terse inet.0: 937054 destinations, 3246024 routes (936878 active, 61 holddown, 419 hidden) Restart Complete + = Active Route, - = Last Active, * = Both A V Destination P Prf Metric 1 Metric 2 Next hop AS path * V 176.61.0.0/18 B 170 100 0 1299 6830 ? valid >208.80.153.218 208.80.153.213 V B 170 100 22640 3356 6830 ? valid >64.156.73.169 ``` We also had the recent complications from accepting MED inbound from Amazon in eqord and eqiad and how we use it internally. We can wait here for the larger evaluation, but I thought it might be worth discussing here either way as it's part of the equation. From my part I don't believe it is serving us well to tie-break on this attribute. My logic being: * We should ultimately control the policy about where we send outbound traffic * Different carriers have different policies, like this between Lumen and Arelion ** Whatever Lumen may be trying to signal, it's definitely not "use Arelion instead" * We should be doing hot-potato routing from our sites towards the internet ** So the only valid MED comparison would be between two routes from the same carrier at the same site I think it would be reasonable, and give us more control plus make things easier to reason about, to just set MED=0 on all inbound routes so it is not used as a tie-break. Either in //BGP_sanitize_in//, or possibly if we didn't want to inlude peers in //BGP_transit_in//.

Looking at some stats I see we barely use our Lumen transit in Codfw. The reason is that most prefixes we see with the same AS-path length from Arelion and Lumen. But Arelion have MED=0 on the routes they send, and Lumen have various values higher than that, which is getting used as a tie-break, for example: ``` cmooney@re0.cr2-codfw> show route protocol bgp 176.61.3433.2 terse inet.0: 937054 destinations, 3246024 routes (936878 active, 61 holddown, 419 hidden) Restart Complete + = Active Route, - = Last Active, * = Both A V Destination P Prf Metric 1 Metric 2 Next hop AS path * V 176.61.0.0/18 B 170 100 0 1299 6830 ? valid >208.80.153.218 208.80.153.213 V B 170 100 22640 3356 6830 ? valid >64.156.73.169 ``` We also had the recent complications from accepting MED inbound from Amazon in eqord and eqiad and how we use it internally. We can wait here for the larger evaluation, but I thought it might be worth discussing here either way as it's part of the equation. From my part I don't believe it is serving us well to tie-break on this attribute. My logic being: * We should ultimately control the policy about where we send outbound traffic * Different carriers have different policies, like this between Lumen and Arelion ** Whatever Lumen may be trying to signal, it's definitely not "use Arelion instead" * We should be doing hot-potato routing from our sites towards the internet ** So the only valid MED comparison would be between two routes from the same carrier at the same site I think it would be reasonable, and give us more control plus make things easier to reason about, to just set MED=0 on all inbound routes so it is not used as a tie-break. Either in //BGP_sanitize_in//, or possibly if we didn't want to inlude peers in //BGP_transit_in//.