I see at least one thing I can explain.

Data:

• tools-prometheus-03 ip addr 172.16.1.8
• hub.paws.wmcloud.org ip addr 185.15.56.57 (floating IP)
• haproxy neutron port VIP 172.16.1.171 (mapped to the floating IP)
• paws-haproxy-1 ip addr 172.16.0.191 (the VM currently holding the neutron port VIP, and thus receiving the floating IP NAT)

Flow:

• From tools-prometheus-03 we send a TCP SYN to hub.paws.wmcloud.org. Packet1: src: 172.16.1.8 dst: 185.15.56.57.
• This packets is DNAT'ed by neutron l3 agent into the neutron port VIP. Packet1: src: 172.16.1.8 dst: 172.16.1.171.
• paws-haproxy-1 recvs the TCP SYN, and replies with SYN+ACK from the same source address it originally recv'd the SYN. Packet2: src: 172.16.1.171 dst: 172.16.1.8.

This is a typical case of asymmetric routing
Moreover, the cloudvirts have stateful firewalling, and Packet2 wont pass a stateful firewall check.

I found this nice diagram on the internet, and I think it's right:

So a simple solution here is to connect to the neutron port VIP directly, instead of the floating IP address. Didn't we have a trick in the resolver to workaround this? Or did we drop it?

In T287107#7229499, @aborrero wrote:

So a simple solution here is to connect to the neutron port VIP directly, instead of the floating IP address. Didn't we have a trick in the resolver to workaround this? Or did we drop it?

We do (labsaliaser), but this address is not directly assigned to a VM (it's managed with keepalived), so it does not find the address when looping thru all instances and their addresses from Nova.

In T287107#7229504, @Majavah wrote:

In T287107#7229499, @aborrero wrote:

So a simple solution here is to connect to the neutron port VIP directly, instead of the floating IP address. Didn't we have a trick in the resolver to workaround this? Or did we drop it?

We do (labsaliaser), but this address is not directly assigned to a VM (it's managed with keepalived), so it does not find the address when looping thru all instances and their addresses from Nova.

so perhaps the simple solution is to add the VIP to the DNS.

In T287107#7229568, @aborrero wrote:

so perhaps the simple solution is to add the VIP to the DNS.

Unfortunately, this is host-based routing and TLS. We know we can fake that out by adding a Host header, but we are limited by what we can do in prometheus. The VIP is in DNS, but I may not be able to scrape it at that address.

We could teach the ip-aliaser to read the connection to a port. Is that what you mean? I can go mess with that :)

[bstorm@cloudcontrol1003]:~ $ sudo wmcs-openstack --os-project-id paws floating ip list --project paws
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+---------+
| ID                                   | Floating IP Address | Fixed IP Address | Port                                 | Floating Network                     | Project |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+---------+
| 1b287d46-acf5-4978-8b4b-4da8cff0ea0b | 185.15.56.57        | 172.16.1.171     | 9c0a9a13-e409-49de-9ba3-bc8ec4801dbf | 5c9ee953-3a19-4e84-be0f-069b5da75123 | paws    |

Just imagine this situation:

user@tools-prometheus-05:~$ dig +short hub.paws.wmcloud.org
172.16.1.171

instead of the current:

user@tools-prometheus-05:~$ dig +short hub.paws.wmcloud.org
185.15.56.57

TLS stuff wont even notice.

Change 706814 had a related patch set uploaded (by Bstorm; author: Bstorm):

[operations/puppet@production] cloud dns: alias all attached floating_ips, not just server ones

https://gerrit.wikimedia.org/r/706814

I realized, the aliaser should totally handle this.

Change 706814 merged by Bstorm:

[operations/puppet@production] cloud dns: alias all attached floating_ips, not just server ones

https://gerrit.wikimedia.org/r/706814

That seems to have worked:

bstorm@tools-prometheus-03:~$ dig @208.80.154.143 hub.paws.wmcloud.org

; <<>> DiG 9.11.5-P4-5.1+deb10u5-Debian <<>> @208.80.154.143 hub.paws.wmcloud.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1609
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;hub.paws.wmcloud.org.		IN	A

;; ANSWER SECTION:
hub.paws.wmcloud.org.	1054	IN	CNAME	paws.wmcloud.org.
paws.wmcloud.org.	3274	IN	A	172.16.1.171