What are the real pros and cons on this? We could even go in the other direction and have a unique ASN per region/continent. How does the impact future anycasting? Note https://tools.ietf.org/html/rfc6382 talks about best practice for anycast being to have distinct ASNs per region, but I don't pretend to understand all the finer details and arguments in that RFC.

In T167840#3346810, @BBlack wrote:

What are the real pros and cons on this? We could even go in the other direction and have a unique ASN per region/continent. How does the impact future anycasting? Note https://tools.ietf.org/html/rfc6382 talks about best practice for anycast being to have distinct ASNs per region, but I don't pretend to understand all the finer details and arguments in that RFC.

I only skimmed over that RFC, but I suspect it's talking about disjoint islands of anycasted services (e.g. DNS root servers). In our case, it's all one single network, with backhauling/transport links and an IGP running all across it, so it may not be very applicable to us.

In practical terms, merging our ASNs simplifies our network a little bit, as well as opens up the door for opportunities to route European-directed traffic from and to the core datacenters over our private datacenter links.

If you really think of it more broadly, we really operate two different networks with different requirements:

• a "core" network, that hosts misc services and fundraising and WMCS etc. with a network that spans the globe and all of our regional PoPs. The routes of the core network could be announced by every PoP and in the other direction, the core network would be able to leverage the global network to route traffic to destinations over its private links.
• a CDN network, every PoP of which is really supposed to serve local, regional traffic. It has separate address spaces per region, to be announced only regionally and never supposed to cross regional boundaries, but needs to leverage the core network for backhauling.

Building those two networks segregated like that would very complicated (e.g. renumbering eqiad/codfw's IP space into separate regional address spaces, VRFs or separate virtual routers for doing the same for outgoing traffic etc.) and a waste of time and resources for us at this point, I think.

The next best scenario IMHO is to just merge our communities and our ASNs for simplicity, and announce "customer" routes (e.g. WMF HQ, soon WMCS, possibly fundraising) across regional boundaries, while keeping 208.80.152.0/22 announced mostly in the US East, or potentially other regions only over transits that support BGP MED.

How I understand it, the increased complexity of running two "networks" outweighs its advantages. And our customers are networks we manage and have control over.
A single ASNs (using confederation) vs. one ASN per region also makes peering relationships easier.
This RFC recommends the use of individual ASN for anycast to be able to make more fine grained routing decisions, which we can also do using communities.
We can also look at similar networks, CDNs, and such, that all operate a single ASN.

To keep the conversation moving, here is a coarse plan to move ESAMS to AS 14907 with little downtime, not tested in a lab, only thoughts. It also includes changes related to T167841 and T83037 to have esams as a model for the other pops, and then core confederations.
Routing policies have been reviews and should be compatible with those changes.

Ideally, to be done after cr2-knams is decommissioned to reduce the number of moving parts but this is not a blocker.

EDIT: Action plan moved to description

I guess there's something to be said for using different ASNs for core vs CDN in the case of losing our transport connectivity to (one of) the CDN sites. We could then still tunnel this over the Internet (IP transits on both sides), which would be much harder (but not impossible) to do with one global ASN.

Mentioned in SAL (#wikimedia-operations) [2017-09-25T19:58:14Z] <XioNoX> renumbering ams BGP communities - T167840

Mentioned in SAL (#wikimedia-operations) [2017-09-27T16:27:25Z] <XioNoX> setting local-as to selected transit BGP sessions - T167840

It seems like Junos' local-as feature isn't working as expected.
Global AS of 43821, remote side with peer-as 43821, and the local side with:
local-as 14907 -> BGP session doesn't establish
local-as 14907 private -> BGP session doesn't establish
local-as 14907 alias -> BGP session establish, AS path is 43821 I (as expected per the doc)

Global AS of 43821, remote side with peer-as 14907, and the local side with:
local-as 14907 -> BGP session establish, AS path is 14907 I, should be 43821 14907 I
local-as 14907 private -> BGP session establish, AS path is 14907 I, should be 43821 I
local-as 14907 alias -> BGP session establish, AS path is 14907 I (as expected per the doc)

The main advantage that this feature would have given us is to advertise prefixes from a single AS# during the transition period, and flip all the advertisements at once to 14907.
Instead, ams prefixes will originate from both AS# until all the providers are switches over to peer with 14907.
This makes things a tad more confusing, but doesn't impact connectivity.
On the upside it allows us to progressively test that all providers allow our ams prefixes to originate from the new AS#.

Change 383382 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] Depool esams for expected blips during ASN renumbering

https://gerrit.wikimedia.org/r/383382

Change 383382 merged by Ayounsi:
[operations/dns@master] Depool esams for expected blips during ASN renumbering

https://gerrit.wikimedia.org/r/383382

Mentioned in SAL (#wikimedia-operations) [2017-10-10T17:32:37Z] <XioNoX> depooling esams from DNS for T167840

After depooling esams the Telia link in eqiad started to saturate, I added the following terms to temporary ease out that link.

[edit policy-options as-path-group AVOID-PATHS]
+    as-path FT-Telia "1299 5511 .*";
+    as-path TelItalia-Telia "1299 6762 .*";

Confirmed working based on interface graph.

Mentioned in SAL (#wikimedia-operations) [2017-10-10T19:00:28Z] <XioNoX> starting the work for T167840

Mentioned in SAL (#wikimedia-operations) [2017-10-10T21:14:59Z] <bblack> esams repooling - T167840

Temporary AVOID-PATHS removed on cr2-eqiad.

The maintenance is now completed, some notes:

• It was not clear that the plan included removing OSPF on the trans-atlantic link in favor of BGP

Need to discuss and familiarize it more with the team

• The BGP filters were not allowing internal routes (10/8) to propagate to AMS, which caused a barrage of alerts.

This could have been avoided by comparing the routes learned via OSPF and the ones learned by BGP after migrating the 1st link, but before migrating the 2nd link

• For the two reasons above, OSPF has been re-established

Communication has been sent to our peers and transits so they update their peer AS#. We can monitor and decide later on when to remove the local-as as well as old v6 IP.

It's not often that one of our primary cache PoPs ends up depooled for multiple hours. While obviously unintended, this was an interesting opportunity to measure the difference Esams makes for web performance.

https://grafana.wikimedia.org/dashboard/db/navigation-timing?from=1507482000000&to=1507680000000
https://grafana.wikimedia.org/dashboard/db/navigation-timing-by-platform?from=1507482000000&to=1507680000000
https://grafana.wikimedia.org/dashboard/db/navigation-timing-by-country?from=1507626000000&to=1507680000000

Navigation Timing: responseStart	NavigationTiming: loadEvent
Time to first byte on page views. Basically 1 round of DNS+TCP+SSL and HTTP response headers.	Time for page to completely finish loading (HTML, CSS, Images)
+100ms (Median of all)	+0.3s (Median of all)
+115ms (Median of mobile)	+0.2s (Median of mobile)
+70ms – +100ms (Median of United_Kingdom)
+90ms – +170ms (Median of Germany)
+68ms – +190ms (Median of Italy)
+60ms – +208ms (Median of Japan)
+83ms – +222ms (Median of Russia)

As of today, 180 BGP sessions use the old AS# and 216 use the new one.

Timeline for decommissioning the old AS# (dates are flexible):

• October 24th, send another round of emails of peers still peering with our old ASN, mentioning the following:
• October 31st (1 week later), Send another round of emails, delete the old IPv6

This will cause the peers using our old v6 IP to drop, without impacting v4 peering, hopefully they will notice it and update their configuration

• November 7th (1 week later), dend another round of emails, remove the "local-as" statement

Will cause all the v4 peers using our old AS# to drop
This can be postponed if the number of peers is still too high, but if they haven't reconfigured their session after 3 weeks, 4 direct emails and a down v6 session I don't think more time will help.

+1 LGTM!

Sounds fine to me. Before we resolve this task, let's not forget that we'll need to cleanup a) our RIPE objects by remove the old route(6) ones b) our RPKI ROAs.

Did AS1126:

maartend@vancis-asd01-r01> show bgp summary | match 14907
80.249.209.176 14907 6 9 0 0 1:49 Establ
2001:7f8:1::a501:4907:1 14907 6 6 0 0 1:45 Establ

More info at http://www.as1126.net/ . Thanks for peering with us.

I was updating RIPE db and I noticed some of the records are still lagging.

• Old AS record is at https://apps.db.ripe.net/search/query.html?searchtext=AS43821%20&bflag=true&source=RIPE#resultsAnchor , it contains from AS43821:AS-PEERS-AMS-IX accept ANY AND NOT {0.0.0.0/0}
• You probably want to rename https://apps.db.ripe.net/search/query.html?searchtext=AS43821:AS-PEERS-AMS-IX&bflag=true&source=RIPE#resultsAnchor to something else
• And update https://apps.db.ripe.net/search/lookup.html?source=ripe&key=AS14907&type=aut-num to use it like AS43821

For bonus points you can also add the route-servers, see https://apps.db.ripe.net/search/query.html?searchtext=AS6777&bflag=true&source=RIPE#resultsAnchor

Mentioned in SAL (#wikimedia-operations) [2017-10-31T21:00:22Z] <XioNoX> removing old AMS-IX IPv6 - T167840

Mentioned in SAL (#wikimedia-operations) [2017-12-19T21:51:18Z] <XioNoX> removing local-as AS43821 from ams transits - T167840