Change Details

> Every single node in the old network design peers with the core routers. When traffic needs to leave a Kubernetes node and go to a Kubernetes node in the same rack/rack row, it needs to go via the core router. This will eventually needlessly saturate switch upstream links. Intra rack/rack row traffic should stay local to the rack/ rack row. This can be achieved easily by route reflectors, which calico supports. It should be pointed out that the new network design does not suffer from this issue and every ToR is effectively a route reflector. The generic industry compliant terminology for the above "deficit" would probably be that East-West traffic gets transformed to North-South. The lack of optimal path happens for any traffic: * towards a prefix advertised by a BGP speaking host in one of the old (`L2` only) rows (so advertised to the core routers directly), which means k8s but also LVS, DNS, etc) * from a host in the same row as that advertising host In the new network design, as the router is the switch (so first device the host connects to), traffic will always flow in an optimal manner. All the "regular" hosts in our DCs don't have any other choice than to talk to their default gateway. So not much can be realistically done here. However all the BGP speaking hosts can potentially receive additional routing information from their peers (usually core routers or ToR switches). We currently have a policy to not send prefixes to BGP speaking servers. They exclusively use their default gateway for outbound traffic. Mostly a tradeoff between simplicity (explicit traffic flows, lean server config, easier troubleshooting) and additional traffic on those uplinks (we can ignore the latency difference here). Especially in the past were we had little BGP speaking hosts. This is obviously changing with k8s. Calico's Bird (BGP daemon) config is not as configurable as a regular Bird daemon, but allows any inbound prefixes by default. That means we can fully decide on what prefixes those hosts can received in the core routers export policies but if additional "mangling" is needed on the host it might make such change much more complex. The change I want to experiment with here (on the staging clusters) is to have the core routers advertise some prefixes back to the k8s hosts. The easiest filtering would be based on source ASNs. There is no easy/clean way to filter on "same row only" source hosts. This solution shouldn't have any additional downsides than the current status-quo (eg. BGP failure detection, etc) as the core routers are still their default gateway (worse case traffic will flow back through the routers). Behaviors such as anycast would need to be looked at more closely when the time comes. If there is lots of traffic from k8s hosts to LVS, we could also let this kind of traffic flow directly. How to prioritize it depends on how much traffic we're talking about, as well as the complexity of the solution. Depending on those two factors it could be fine to wait for the new design to be progressively rolled out everywhere (but we're talking 2 years here).

> Every single node in the old network design peers with the core routers. When traffic needs to leave a Kubernetes node and go to a Kubernetes node in the same rack/rack row, it needs to go via the core router. This will eventually needlessly saturate switch upstream links. Intra rack/rack row traffic should stay local to the rack/ rack row. This can be achieved easily by route reflectors, which calico supports. It should be pointed out that the new network design does not suffer from this issue and every ToR is effectively a route reflector. The generic industry compliant terminology for the above "deficit" would probably be that East-West traffic gets transformed to North-South. The lack of optimal path happens for any traffic: * towards a prefix advertised by a BGP speaking host in one of the old (`L2` only) rows (so advertised to the core routers directly), which means k8s but also LVS, DNS, etc) * from a host in the same row as that advertising host In the new network design, as the router is the switch (so first device the host connects to), traffic will always flow in an optimal manner. All the "regular" hosts in our DCs don't have any other choice than to talk to their default gateway. So not much can be realistically done here. However all the BGP speaking hosts can potentially receive additional routing information from their peers (usually core routers or ToR switches). We currently have a policy to not send prefixes to BGP speaking servers. They exclusively use their default gateway for outbound traffic. Mostly a tradeoff between simplicity (explicit traffic flows, lean server config, easier troubleshooting) and additional traffic on those uplinks (we can ignore the latency difference here). Especially in the past were we had little BGP speaking hosts. This is obviously changing with k8s. Calico's Bird (BGP daemon) config is not as configurable as a regular Bird daemon, but allows any inbound prefixes by default. That means we can fully decide on what prefixes those hosts can received in the core routers export policies but if additional "mangling" is needed on the host it might make such change much more complex. The change I want to experiment with here (on the staging clusters) is to have the core routers advertise some prefixes back to the k8s hosts. The easiest filtering would be based on source ASNs. There is no easy/clean way to filter on "same row only" source hosts. This solution shouldn't have any additional downsides than the current status-quo (eg. BGP failure detection, etc) as the core routers are still their default gateway (worse case traffic will flow back through the routers). Behaviors such as anycast would need to be looked at more closely when the time comes. If there is lots of traffic from k8s hosts to LVS, we could also let this kind of traffic flow directly. How to prioritize it depends on how much traffic we're talking about, as well as the complexity of the solution. Depending on those two factors it could be fine to wait for the new design to be progressively rolled out everywhere (but we're talking 2 years here). EDIT, putting some more thoughts into it: BGP is smart about it (see '"first party" NEXT_HOP' in section 5.1.3.2 of the [[ https://www.rfc-editor.org/rfc/rfc4271#section-5.1.3.2 | RFC ]]), so it **should** just work on the router side. However using Calico's `numAllowedLocalASNumbers` [[ https://projectcalico.docs.tigera.io/reference/resources/bgppeer | config knob ]] will be needed, as all the nodes from a given cluster use the same AS#.

> Every single node in the old network design peers with the core routers. When traffic needs to leave a Kubernetes node and go to a Kubernetes node in the same rack/rack row, it needs to go via the core router. This will eventually needlessly saturate switch upstream links. Intra rack/rack row traffic should stay local to the rack/ rack row. This can be achieved easily by route reflectors, which calico supports. It should be pointed out that the new network design does not suffer from this issue and every ToR is effectively a route reflector. The generic industry compliant terminology for the above "deficit" would probably be that East-West traffic gets transformed to North-South. The lack of optimal path happens for any traffic: * towards a prefix advertised by a BGP speaking host in one of the old (`L2` only) rows (so advertised to the core routers directly), which means k8s but also LVS, DNS, etc) * from a host in the same row as that advertising host In the new network design, as the router is the switch (so first device the host connects to), traffic will always flow in an optimal manner. All the "regular" hosts in our DCs don't have any other choice than to talk to their default gateway. So not much can be realistically done here. However all the BGP speaking hosts can potentially receive additional routing information from their peers (usually core routers or ToR switches). We currently have a policy to not send prefixes to BGP speaking servers. They exclusively use their default gateway for outbound traffic. Mostly a tradeoff between simplicity (explicit traffic flows, lean server config, easier troubleshooting) and additional traffic on those uplinks (we can ignore the latency difference here). Especially in the past were we had little BGP speaking hosts. This is obviously changing with k8s. Calico's Bird (BGP daemon) config is not as configurable as a regular Bird daemon, but allows any inbound prefixes by default. That means we can fully decide on what prefixes those hosts can received in the core routers export policies but if additional "mangling" is needed on the host it might make such change much more complex. The change I want to experiment with here (on the staging clusters) is to have the core routers advertise some prefixes back to the k8s hosts. The easiest filtering would be based on source ASNs. There is no easy/clean way to filter on "same row only" source hosts. This solution shouldn't have any additional downsides than the current status-quo (eg. BGP failure detection, etc) as the core routers are still their default gateway (worse case traffic will flow back through the routers). Behaviors such as anycast would need to be looked at more closely when the time comes. If there is lots of traffic from k8s hosts to LVS, we could also let this kind of traffic flow directly. How to prioritize it depends on how much traffic we're talking about, as well as the complexity of the solution. Depending on those two factors it could be fine to wait for the new design to be progressively rolled out everywhere (but we're talking 2 years here). EDIT, putting some more thoughts into it: BGP is smart about it (see '"first party" NEXT_HOP' in section 5.1.3.2 of the [[ https://www.rfc-editor.org/rfc/rfc4271#section-5.1.3.2 | RFC ]]), so it **should** just work on the router side. However using Calico's `numAllowedLocalASNumbers` [[ https://projectcalico.docs.tigera.io/reference/resources/bgppeer | config knob ]] will be needed, as all the nodes from a given cluster use the same AS#.