The issue is that under some circumstances, Neutron can have 2 active routing nodes (l3-agent in cloudnet boxes), while our current configuration only supports the active-passive model.
This results in duplicated IP addresses in the network (neutron gw), asymmetric routing, duplicated packets, and other issues.
Trigger is usually a simple puppet change, to some neutron component because currently puppet agent restarts the l3-agent. The restarted l3-agent tries to go directly to the active role within neutron, even if there is already another l3-agent in active role.
At first we were thinking this was just some nova-network<->neutron compat network thing, but eqiad1-r hosts are behaving strangely over the internet.