We set up the new paws entrypoint using keepalived instead of the usual static and manually moved floating IP. This is good. However, we never taught labsaliaser about this new thing, so inside cloud you cannot hit https://hub.paws.wmcloud.org/hub/metrics (for example). It should either be able to map to its internal IP (which I believe is what k8s.svc.paws.eqiad1.wikimedia.cloud uses) or perhaps we should fix the bug that requires labsaliaser?
Either way, it seems to be a thing to fix if we start making auto-failover more common.
Update: this seems to be an issue when routing from a VM to a neutron floating IP in a different VM. such as we use in this setup. There is an asymmetric packet flow, moreover, the reply packet doesn't use NAT at all.
This diagram should help understand the issue.