From T263674.
Current status:
pfw3-codfw sends security syslog (over TCP) successfully to both eqiad and codfw collectors
pfw3-eqiad sends security syslog (over TCP) successfully to only codfw
eqiad doesn't work.
pfw3-eqiad> restart security-log gracefully
Sep 24 16:04:12 pfw3-eqiad RT_SYSTEM: RTLOG_CONN_OPEN: Connection established syslog-tls-stream-codfw TLS 10.64.40.65/12432 10.195.0.76/6514
Sep 24 16:04:23 pfw3-eqiad RT_SYSTEM: RTLOG_CONN_ERROR: Connection error syslog-tls-stream Com 12429 abort
Sep 24 16:04:23 pfw3-eqiad RT_SYSTEM: RTLOG_CONN_ERROR: Connection error syslog-tls-stream Error code: major 3 minor 1 code 110, description:TCP time out after SYN is sent out
Sep 24 16:04:23 pfw3-eqiad RT_SYSTEM: RTLOG_CONN_ERROR: Connection error syslog-tls-stream status: 0, Error code: major 3 minor 1 code 110, description:TCP time out after SYN is sent out
Using tcpdump on frlog1001 (which is in the same vlan as the pfw3 interface initiating the handshake), we can see the pfw3 sending the intial SYN, then frlog1001 sending back a SYN-ACK, and then both re-trying.
On the pfw3 side, we see the following:
15:54:35.742502 In IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto: TCP (6), length: 44) 10.64.40.72.6514 > 10.64.40.65.12118: S 1037813821:1037813821(0) ack 2167186818 win 29200 <mss 1460>
So not sure why it's not registering the SYN-ACK.
Maybe a pfw3 bug?