Investigate internal rejected prefixes
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ayounsi
	Apr 15 2026, 8:38 AM

Project Tags

Referenced Files

None

Subscribers

Description

Thanks to T356877: Increase visibility of kubernetes network status I realized that we probably should keep an eye on the prefixes that we reject internally.

On a case by case basis we should then edit the outbound filters, so we never need to reject a prefix on the inbound side.

That way we could have a generic Netops alert to trigger when we start rejecting prefixes.
If that alert triggers too much because of services on the various hosts, we could then offload that alert to service owners.

For example with https://grafana.wikimedia.org/goto/dfj5c1kceij9cc?orgId=1
gnmi_bgp_neighbor_prefixes_rejected{peer_group!~"(IX|Private-Peer|Transit)[4|6]"} > 0
or
gnmi_bgp_neighbor_prefixes_received_pre_policy{peer_group!~"(IX|Private-Peer|Transit)[4|6]"} - gnmi_bgp_neighbor_prefixes_received{peer_group!~"(IX|Private-Peer|Transit)[4|6]"} > 0

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
Add RejectingBGPPrefixes alert	operations/alerts	master	+37 -1
Nokia: add missing Wikidough prefix	operations/homer/public	master	+4 -0
Nokia: also allow anycast prefixes from Ganeti peers	operations/homer/public	master	+23 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		ayounsi	T423384 Investigate internal rejected prefixes
		Open		cmooney	T423430 Don't announce OSPF routes in unicast BGP on Nokia SR-Linux

Event Timeline

ayounsi created this task.Apr 15 2026, 8:38 AM

ayounsi triaged this task as Low priority.

Restricted Application added a project: Infrastructure-Foundations. · View Herald TranscriptApr 15 2026, 8:38 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

ayounsi mentioned this in T356877: Increase visibility of kubernetes network status.Apr 15 2026, 8:39 AM

cmooney added a subtask: T423430: Don't announce OSPF routes in unicast BGP on Nokia SR-Linux.Apr 15 2026, 2:53 PM

Change #1294931 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Nokia: also alow anycast prefixes from Ganeti peers

https://gerrit.wikimedia.org/r/1294931

gerritbot added a project: Patch-For-Review.Thu, May 28, 8:40 AM

{
address="10.128.0.20",
afi_safi="IPV4_UNICAST",
instance="asw1-22-ulsfo:9804",
job="gnmi",
network_instance_name="default",
peer_as="64612",
peer_descr="ganeti4005",
peer_group="ganeti4",
peer_type="EXTERNAL",
prometheus="ops",
protocol_identifier="BGP",
protocol_name="BGP",
site="ulsfo"
}

Confirmed on the switches:

asw1-22-ulsfo# info from state / network-instance default bgp-rib afi-safi ipv4-unicast ipv4-unicast rib-in-out rib-in-post route 10.3.0.10/32 neighbor 10.128.0.20 path-id 0
[...]
rejected-route true

Will be fixed with: https://gerrit.wikimedia.org/r/c/operations/homer/public/+/1294931

Mentioned in SAL (#wikimedia-operations) [2026-05-28T08:50:09Z] <XioNoX> cr1-codfw# delete protocols bgp group fundraising family inet6 - T423384

Another one was:

pfw1-codfw> show route receive-protocol bgp 208.80.153.202 hidden extensive    

inet.0: 26 destinations, 29 routes (26 active, 0 holddown, 0 hidden)
Restart Complete

inet6.0: 1 destinations, 1 routes (0 active, 0 holddown, 1 hidden)
Restart Complete
  ::/0 (1 entry, 0 announced)
     Nexthop: ::ffff:208.80.153.202
     MED: 100
     AS path: 14907 I 
     Hidden reason: Protocol nexthop is not on the interface

Cleared with pfw1-codfw# delete protocols bgp group Production family inet6 as the pfw doesn't have IPv6. Done the same on the cr side to keep it clean.

Another one, pfw1 re-advertises its uplinks subnets to cr1/2-codfw:

cr2-codfw# run show route receive-protocol bgp 208.80.153.203 hidden extensive 

inet.0: 1042192 destinations, 3688335 routes (1041891 active, 0 holddown, 912 hidden)
Restart Complete
  208.80.153.200/31 (2 entries, 1 announced)
     Nexthop: 208.80.153.203
     MED: 100
     AS path: 64701 I 
     Hidden reason: Rejected by import policy

  208.80.153.202/31 (2 entries, 1 announced)
     Nexthop: 208.80.153.203
     MED: 100
     AS path: 64701 I 
     Hidden reason: Rejected by import policy

I'm suggesting we add this term on the pfw side:

[edit policy-options policy-statement BGP_fundraising_export]
     term no_management { ... }
+    term no_uplinks {
+        from interface [ xe-0/2/0.0 xe-7/2/0.0 ];
+        then reject;
+    }
     term connected { ... }

Full policy:

term no_management {
    from interface fxp0.0;
    then reject;
}
term no_uplinks {
    from interface [ xe-0/2/0.0 xe-7/2/0.0 ];
    then reject;
}
term connected {
    from protocol direct;
    then accept;
}
term nat {
    from {
        protocol aggregate;
        prefix-list fundraising-codfw-external4;
    }
    then accept;
}
then reject;

An alternative to not have to harcode interfaces names would be to reject directly connected /31s, which would also block the internal IPsec tunnel range, but I don't think it's an issue.

+    term no_p2p {
+        from {
+            protocol direct;
+            route-filter 0.0.0.0/0 prefix-length-range /31-/31;
+        }
+        then reject;
+    }