The scenario we need to prevent is the following:
- User establish a TCP session to an anycasted VIP: packets are for example flowing through: user -> transit_A -> cr1 -> server1
- A routing change on the Internet happen and the user path becomes: user -> transit_B -> cr2 -> server2
This would cause the TCP session to break as server2 has no knowledge of that client.
The two possible ways for the infra to handle that use case are:
- A) user -> transit_B -> cr2 -> server1 consistently
- B) user -> transit_B -> cr2 -> cr1 -> server1
As hashing is done on source/dest L3 headers, the cr1->server1 step is consistent.
About A) I so far can't find any mention of it in Juniper's consistent hashing doc (here and there). Which mean we would need to test it to figure out if it works as we want it. Which is unlikely.
Regardless, Juniper consistent LB only works with single-hop eBGP, while we currently do multi-hop (as the servers peer with the routers loopback). And only on MPCs, so maybe not on MX204s.
As a side note, Juniper's consistent LB means that if the server pool is >2, adding/removing a server will not reshuffle all the sessions, which we don't need to care about right now as we only have 2 servers per sites.
However, it is possible to use BGP MEDs to achieve B). By having all servers tag a higher MED to a prefix when they advertise that prefix to the same router.
Edit: according to https://apps.juniper.net/feature-explorer/feature-info.html?fKey=6434&fn=Consistent%20load%20balancing%20for%20ECMP%20groups consistent-hash is available on MX204s.