Thanks for the report.

This is because we advertise our "customer" prefixes from all our POPs to improve the user experience by transiting through our network instead of over the Internet.

As you pointed out there are edge cases, and perhaps those edge cases alone justify this type of routing ("cold potato"). This might also make our configuration simpler.

We will discuss it with @cmooney and get back to you.

Indeed thanks for the report.

@ayounsi overall I would agree with us making this change, at least for the WMCS Eqiad prefix (185.15.56.0/24).

Customer-originated prefixes should probably not be announced to the internet except for at the POPs where we receive them. While one might argue our own transport is more reliable, or at least we have eyes on it, it's likely to always incur a latency penalty for users. On balance I think it's better the customer goes internet -> destination pop.

The other question is if we should distribute them internally between POPs, i.e. what path our own servers in Eqsin would use to get to WMCS Eqiad or similar. I'm inclined to say we should keep that over our own transport, but happy to discuss.

Change 947993 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Only advertise local customers to external peers

https://gerrit.wikimedia.org/r/947993

@cmooney mentioned me that the previous syntax didn't work, this is because the as-path-calc-length term ignores confederations.

I updated the patch and tested it in ulsfo, this works as expected:

cr3-ulsfo# run show route advertising-protocol bgp 4.15.72.113 
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 185.15.56.0/24          Self                                    ?
* 185.71.138.0/24         Self                                    I
* 198.35.26.0/24          Self                                    I
* 198.35.27.0/24          Self                                    I
* 198.73.209.0/24         Self                                    11820 ?

[edit policy-options policy-statement BGP_outfilter term customers from]
+     protocol bgp;
+     as-path only_one_asn;

cr3-ulsfo# run show route advertising-protocol bgp 4.15.72.113
  Prefix		  Nexthop	       MED     Lclpref    AS path
* 185.71.138.0/24         Self                                    I
* 198.35.26.0/24          Self                                    I
* 198.35.27.0/24          Self                                    I
* 198.73.209.0/24         Self                                    11820 ?

Change 947993 merged by jenkins-bot:

[operations/homer/public@master] Only advertise local customers to external peers

https://gerrit.wikimedia.org/r/947993