Mentioned in SAL (#wikimedia-operations) [2024-04-17T16:56:34Z] <topranks> running authdns-update to make magru dns records live T362421

Change #1020901 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Remove comment added in error

https://gerrit.wikimedia.org/r/1020901

Change #1020901 merged by Cathal Mooney:

[operations/dns@master] Remove comment added in error

https://gerrit.wikimedia.org/r/1020901

Hi, after 73470d0dca68abee0 ntp no longer auto-restarts, but after one of the latest changes (I believe b48874a81565b7051be39659c056), it is pending. Can it be restarted or should it be kept with the old config for a while, and it should be acked?

In T362421#9725493, @jcrespo wrote:

Hi, after 73470d0dca68abee0 ntp no longer auto-restarts, but after one of the latest changes (I believe b48874a81565b7051be39659c056), it is pending. Can it be restarted or should it be kept with the old config for a while, and it should be acked?

Hmm that's a good point. @ssingh can you comment here? I'm wondering about the original change, why would it matter if we restart the NTP service "quickly" if we have a new set of peers? The system clock is unlikely to drift by much during the restart if it was synced previously? I take it we had some issue though.

Thanks @jcrespo! I should have silenced the alert or restarted the service; both of those are in progress now so we should see this resolve soon.

@cmooney: Previously we were letting Puppet restart the ntp service within the 30-minute or so window it takes to roll out all changes. On the core sites, it takes around ~10 minutes for the NTP sync to be established with the public pools and other hosts, so that's one thing and since we couldn't figure out a reasonable way within Puppet to splay the restarts, we decided to do this manually instead of letting Puppet do it whenever it wanted. And then we also wanted to fix the issue of when there was a new hardware commissioned and for the initial sync and again basically being in control of when we restart the NTP daemon on the various boxes. So we no longer let Puppet do it, but we have this alert that reminds us that we have to. You are right that the system clock is likely to be in sync during regular restarts anyway but this is more about the entire cluster.

Change #1020087 merged by jenkins-bot:

[operations/cookbooks@master] Add configuration for the new magru DC

https://gerrit.wikimedia.org/r/1020087

Change #1019292 merged by jenkins-bot:

[operations/homer/public@master] Add magru to homer-public

https://gerrit.wikimedia.org/r/1019292

In T362421#9710346, @ayounsi wrote:

Prefixes assigned in Netbox: https://netbox.wikimedia.org/ipam/prefixes/?site_id=11

Thanks!

Next step is to create the devices in Netbox and assign the IPs to the interfaces.

Done. All configs now generate fine with Homer \o/

INFO:homer:Homer run completed successfully on 5 devices: ['asw1-b3-magru.mgmt.magru.wmnet', 'asw1-b4-magru.mgmt.magru.wmnet', 'cr1-magru.wikimedia.org', 'cr2-magru.wikimedia.org', 'mr1-magru.wikimedia.org']

Change #1021920 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Add dummy IPs and uncomment vars for magru

https://gerrit.wikimedia.org/r/1021920

Change #1021920 merged by jenkins-bot:

[operations/homer/public@master] Add dummy IPs and uncomment vars for magru

https://gerrit.wikimedia.org/r/1021920

Change #1021967 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Set magru DHCP relay server to install1004

https://gerrit.wikimedia.org/r/1021967

Change #1021967 merged by jenkins-bot:

[operations/homer/public@master] Set magru DHCP relay server to install1004

https://gerrit.wikimedia.org/r/1021967

Change #1022098 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Reverses for 3 new network connections in magru

https://gerrit.wikimedia.org/r/1022098

Icinga downtime and Alertmanager silence (ID=2c797c95-485f-45b4-85c7-e8514173ae11) set by cmooney@cumin1002 for 0:20:00 on 4 host(s) and their services with reason: disabling oob link on mr1-ulsfo to stop the SSH attempts long enough to get a homer run in

mr1-ulsfo,mr1-ulsfo IPv6,mr1-ulsfo.oob,mr1-ulsfo.oob IPv6

Change #1022098 merged by Cathal Mooney:

[operations/dns@master] Reverses for 3 new network connections in magru

https://gerrit.wikimedia.org/r/1022098

Change #1024516 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] magru: update edgeuno transit IP

https://gerrit.wikimedia.org/r/1024516

Change #1024516 merged by jenkins-bot:

[operations/homer/public@master] magru: update edgeuno transit IP

https://gerrit.wikimedia.org/r/1024516

Change #1024815 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] magru: add momentum/novacore peer IPs/AS

https://gerrit.wikimedia.org/r/1024815

Change #1024815 merged by jenkins-bot:

[operations/homer/public@master] magru: add momentum/novacore peer IPs/AS

https://gerrit.wikimedia.org/r/1024815

Change #1024848 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Add AS65007 to confederation

https://gerrit.wikimedia.org/r/1024848

Change #1024848 merged by jenkins-bot:

[operations/homer/public@master] Add AS65007 to confederation

https://gerrit.wikimedia.org/r/1024848

Change #1024894 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Add magru to Rancid

https://gerrit.wikimedia.org/r/1024894

Change #1024895 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Add magru network to monitoring

https://gerrit.wikimedia.org/r/1024895

Change #1024894 merged by Ayounsi:

[operations/puppet@production] Add magru to Rancid

https://gerrit.wikimedia.org/r/1024894

Change #1024895 merged by Ayounsi:

[operations/puppet@production] Add magru network to monitoring

https://gerrit.wikimedia.org/r/1024895

Change #1025414 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] magru: update novacore v6 IP

https://gerrit.wikimedia.org/r/1025414

Change #1025414 merged by jenkins-bot:

[operations/homer/public@master] magru: update novacore v6 IP

https://gerrit.wikimedia.org/r/1025414

Change #1025442 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] magru: update novacore IPv6 once more

https://gerrit.wikimedia.org/r/1025442

Change #1025442 merged by jenkins-bot:

[operations/homer/public@master] magru: update novacore IPv6 once more

https://gerrit.wikimedia.org/r/1025442

Mentioned in SAL (#wikimedia-operations) [2024-05-01T09:22:38Z] <topranks> withdrawing public prefix announcement to AS7195 to test backup in magru (T362421)

Change #1026525 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/dns@master] Add include for svc.magru.wmnet in wmnet zone

https://gerrit.wikimedia.org/r/1026525

Change #1026525 merged by Cathal Mooney:

[operations/dns@master] Add include for svc.magru.wmnet in wmnet zone

https://gerrit.wikimedia.org/r/1026525

Change #1026577 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] magru: add netflow7001 to kafka ACLs

https://gerrit.wikimedia.org/r/1026577

Change #1026577 abandoned by Ayounsi:

[operations/puppet@production] magru: add netflow7001 to kafka ACLs

Reason:

If5fff3828f3687dec4f531f490666d53c1c4fc09

https://gerrit.wikimedia.org/r/1026577

Change #1026578 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] magru: set netflow collector IP

https://gerrit.wikimedia.org/r/1026578

Change #1026578 merged by jenkins-bot:

[operations/homer/public@master] magru: set netflow collector IP

https://gerrit.wikimedia.org/r/1026578

Change #1026808 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] magru: alert on Transit BGP sessions

https://gerrit.wikimedia.org/r/1026808

Change #1026808 merged by Ayounsi:

[operations/puppet@production] magru: alert on Transit BGP sessions

https://gerrit.wikimedia.org/r/1026808

Change #1026956 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Add VM BGP for esams/drmrs/magru back to YAML for now

https://gerrit.wikimedia.org/r/1026956

Change #1026956 abandoned by Cathal Mooney:

[operations/homer/public@master] Add VM BGP for esams/drmrs/magru back to YAML for now

Reason:

will tackle via changes in homer rather than going back to yaml

https://gerrit.wikimedia.org/r/1026956

Change #1032520 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Announce Wikidough Anycast ranges to internet from magru

https://gerrit.wikimedia.org/r/1032520

Change #1032520 merged by jenkins-bot:

[operations/homer/public@master] Announce Wikidough Anycast ranges to internet from magru

https://gerrit.wikimedia.org/r/1032520

Mentioned in SAL (#wikimedia-operations) [2024-05-16T16:12:46Z] <topranks> announcing wikidough anycast ranges to Inernet (transit) in magru T362421

Thanks to @cmooney for rolling the above out. For further context, we (Traffic and netops) decided to try out the anycast range in magru for the Wikidough service before doing it for more critical things, such as announcing the ns2.wikimedia.org IP from magru. The timeline on that still is a TBD but this is a nice test to make sure that the configuration was correct.

And fwiw announcement looks good, all 3 of our transits are learning it ok, and I see it on other carriers from those sources as well. We also see live requests on the doh servers.

Before advertising ns2, we need to do some traffic engineering. Telxius being part of Spain's main ISP, Telefonica ES prefers magru to drmrs :
See https://w.wiki/A6qH

I filtered out some obvious cloud providers that probably just do random internet scans. We will also have better SNR with ns2 online from magru for further tuning.

EdgeUno have BGP communities, if we notice any issues we can stop the propagation towards US/TK/DE : https://edgeuno.com/bgp/
Telxius I emailed them to ask about their BGP communities, but also found https://wiki.brasilpeeringforum.org/w/Lista_de_Communities_BGP#Telxius The informative communities match, so I'll implement the 65033:1 community and see if it behaves as it should. I think this doc has a typo and it will prepend 3x to EU peers). Then probably expand it to the Asia and North America communities.

+1 sounds like a good idea. Nice we have some limited scope to experiment with the DoH ranges before pulling the plug on ns2.

FWIW I think these would be the ones to use with EdgeUno:

64001:63000 - USA Tranist
64001:65000 - USA Peering

64049:63000 - DE Tranist
64049:65000 - DE Peering

64090:63000 - Turkey Tranist
64090:65000 - Turkey Peering

Cogent are picking the magru announcement as best globally from Novvacore it seems also. We could add 28189:8094 to not export it to Cogent at all? Or 28189:215[3|6] to prepend either 3 or 6 times?

Change #1032747 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] magru: 3x prepending for Anycast prefixes to Telxius

https://gerrit.wikimedia.org/r/1032747

Change #1032747 merged by jenkins-bot:

[operations/homer/public@master] magru: 3x prepending for Anycast prefixes to Telxius

https://gerrit.wikimedia.org/r/1032747

Cogent is a bit surprising, from EU or the US they route to magru.

Fri May 17 10:29:23.898 UTC
BGP routing table entry for 185.71.138.0/24
Versions:
  Process           bRIB/RIB  SendTblVer
  Speaker         2566276107  2566276107
Last Modified: May 16 16:16:26.294 for 18:12:57
Paths: (1 available, best #1)
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    38.5.0.99       
  Path #1: Received by speaker 0
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    38.5.0.99       
  28189 14907, (aggregated by 14907 195.200.68.128)
    199.100.0.138 (metric 294060) from 38.28.1.83 (38.28.20.16)
      Origin IGP, localpref 140, valid, internal, best, group-best, import-candidate
      Received Path ID 0, Local Path ID 1, version 2566276107
      Community: 174:140 174:13001 174:20999 174:21301 174:22042
      Originator: 38.28.20.16, Cluster list: 38.28.1.83, 38.28.1.67, 38.28.1.115, 154.54.66.68, 66.28.1.87


Fri May 17 10:24:30.480 UTC
BGP routing table entry for 185.71.138.0/24
Versions:
  Process           bRIB/RIB  SendTblVer
  Speaker          485009198   485009198
Last Modified: May 16 16:09:32.020 for 18:14:58
Paths: (1 available, best #1)
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    38.5.4.117      
  Path #1: Received by speaker 0
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    38.5.4.117      
  28189 14907, (aggregated by 14907 195.200.68.128)
    199.100.0.138 (metric 117050) from 154.54.66.234 (38.28.20.16)
      Origin IGP, localpref 140, valid, internal, best, group-best, import-candidate
      Received Path ID 0, Local Path ID 1, version 485009198
      Community: 174:140 174:13001 174:20999 174:21301 174:22042
      Originator: 38.28.20.16, Cluster list: 154.54.66.234, 66.28.1.31, 66.28.1.87

They might prefer going through EdgeUno once we add the prepending to Novvacore, so the same change would be needed there as well.

In T362421#9808194, @ayounsi wrote:

They might prefer going through EdgeUno once we add the prepending to Novvacore, so the same change would be needed there as well.

It's possible alright. I notice they've got local-pref 140 on that route, presumably as it's from a direct customer and thus preferred. They see the NS2 range from Arelion but pref is 100:

Fri May 17 10:50:26.671 UTC
BGP routing table entry for 198.35.27.0/24
Versions:
  Process           bRIB/RIB  SendTblVer
  Speaker          415731150   415731150
Last Modified: May  1 09:17:29.020 for 2w2d
Paths: (1 available, best #1)
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    38.5.4.117      
  Path #1: Received by speaker 0
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    38.5.4.117      
  1299 14907, (aggregated by 14907 208.80.154.197)
    154.54.12.62 (metric 3030) from 154.54.66.234 (66.28.1.9)
      Origin IGP, metric 4294967294, localpref 100, valid, internal, best, group-best, import-candidate
      Received Path ID 0, Local Path ID 1, version 415731150
      Community: 174:10031 174:20666 174:21000 174:22013
      Originator: 66.28.1.9, Cluster list: 154.54.66.234, 66.28.1.31

We may need to try and prevent either from exporting it to them completely.

Given they seem to select one best path globally Aerlion is probably the best for them to be using, as we have transit from them at multiple sites.

The Telxius community doesn't seem to be of any effect so far, I'll wait for their reply, maybe they changed or need to be enabled on their side first. I'll look at the other providers afterwards.

In T362421#9808627, @ayounsi wrote:

The Telxius community doesn't seem to be of any effect so far, I'll wait for their reply, maybe they changed or need to be enabled on their side first. I'll look at the other providers afterwards.

Is there a looking-glass on TF ES we can use to check? Could just be local-pref trumping as-path. We might need/want to add 65000:3352 to stop them propagating it to them completely?

Or replace the pre-pend 3 times community for regions other than Latin America with the 'no-export' one for them?

Change #1034506 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Magru/Telxius: don't readvertise anycast prefixes to NA/AS/EU

https://gerrit.wikimedia.org/r/1034506

Change #1034506 merged by jenkins-bot:

[operations/homer/public@master] Magru/Telxius: don't readvertise anycast prefixes to NA/AS/EU

https://gerrit.wikimedia.org/r/1034506

Change #1035371 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Enable BFD on Telxius transit

https://gerrit.wikimedia.org/r/1035371

Change #1035371 merged by jenkins-bot:

[operations/homer/public@master] Enable BFD on Telxius transit

https://gerrit.wikimedia.org/r/1035371

In T362421#9808055, @cmooney wrote:

Cogent are picking the magru announcement as best globally from Novvacore it seems also. We could add 28189:8094 to not export it to Cogent at all? Or 28189:215[3|6] to prepend either 3 or 6 times?

I'm not delighted with this to be honest. I'm gonna add that for now as Cogent are still routing to magru via Novvacore globally which we don't really want. We can revise again.

FWIW I tried the pre-pend 6 times towards Cogent on the Novacare link but they are still selecting it:

Thu May 30 16:50:04.801 UTC
BGP routing table entry for 198.35.27.0/24
Versions:
  Process           bRIB/RIB  SendTblVer
  Speaker          550553751   550553751
Last Modified: May 30 16:49:08.020 for 00:00:57
Paths: (1 available, best #1)
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    38.5.4.117      
  Path #1: Received by speaker 0
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    38.5.4.117      
  28189 28189 28189 28189 28189 28189 28189 14907, (aggregated by 14907 195.200.68.128)
    199.100.0.138 (metric 117050) from 154.54.66.234 (38.28.20.16)
      Origin IGP, localpref 130, valid, internal, best, group-best, import-candidate
      Received Path ID 0, Local Path ID 1, version 550553751
      Community: 174:13001 174:20999 174:21301 174:22042
      Originator: 38.28.20.16, Cluster list: 154.54.66.234, 66.28.1.31, 66.28.1.87

local-pref 130 there probably for direct customer's rather than other tranists. I'll change it to 28189:8094 to not announce them at all.

Ok after change Cogent are going to Telia, which we have at all the other POPs, so I think a better result. Novvacore are still announcing it at IX.br so doesn't seem to have affected local propagation.

I'll submit a patch and we can discuss whether to make this permanent or reverse.

Thu May 30 16:55:29.929 UTC
BGP routing table entry for 198.35.27.0/24
Versions:
  Process           bRIB/RIB  SendTblVer
  Speaker          550573876   550573876
Last Modified: May 30 16:54:51.020 for 00:00:39
Paths: (1 available, best #1)
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    38.5.4.117      
  Path #1: Received by speaker 0
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    38.5.4.117      
  1299 14907, (aggregated by 14907 208.80.154.197)
    154.54.12.62 (metric 3030) from 154.54.66.234 (66.28.1.9)
      Origin IGP, metric 4294967294, localpref 100, valid, internal, best, group-best, import-candidate
      Received Path ID 0, Local Path ID 1, version 550573876
      Community: 174:10031 174:20666 174:21000 174:22013
      Originator: 66.28.1.9, Cluster list: 154.54.66.234, 66.28.1.31

Change #1037588 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Policy for Novvacore at magru to not announce Anycast to Cogent

https://gerrit.wikimedia.org/r/1037588

Change #1037736 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] magru: add BGP to HE

https://gerrit.wikimedia.org/r/1037736

Change #1037736 merged by jenkins-bot:

[operations/homer/public@master] magru: add BGP to HE

https://gerrit.wikimedia.org/r/1037736

Change #1037588 merged by jenkins-bot:

[operations/homer/public@master] Policy for Novvacore at magru to not announce Anycast to Cogent

https://gerrit.wikimedia.org/r/1037588