I opened T325806: Help refreshing firewall dashboards to get the dashboard back online, I'll re-open it if there is any need. Thanks!

I looked at this last week, and while I didn't get to the bottom of it, my suspicion here is that the logs are due to a race-condition when the connection terminates.

I specifically was looking at the pcap of the tcp sessions initiated by blazegraph on wqds1013 to eventgate.

It seems to me that the client is configured to always send a RST immediately after the FIN it sends. My speculation is it is doing so to quickly close the state on the server, given the short-lived connections it uses, rather than have many connections in the TIME_WAIT state.

Based on the packet capture for most connections there is no problem, and no error logs. The client connects, sends some data, sends a FIN/ACK and then immediately a RST. The server does not send any packets after the FIN and RST. This is shown in this packet capture:

In some cases, however, the client sends multiple RSTs, and we have errors logged, see the below exchange for an example:

Looks to me like this happens when the server also tries to actively close the connection. My guess is in both scenarios client and server know the exchange is over, and both are ready to close the connection. In most cases the client sends its FIN and RST before the server has done anything, and the server tears things down without any more communication. In some cases, however, I'm guessing the server actually starts to close the connection before it gets the FIN and RST from client. And as per this capture sends a HTTP close (packet 18), then FIN (20) and an ACK to the clients FIN (22).

As the client has already sent a RST, and torn down the connection, there is no conntrack entry for it when these 3 packets reach the client. Which causes them to be logged.

If the client did not send the RST immediately on close, trying to hard-reset the session, this would not happen I think. But that may be desirable behaviour for short lived connections like this. I did spend some time looking at the various code-bases involved to find where this might be configured, I believe this is the behvaiour in Linux if SO_LINGER is set to 0 for the connection. But I didn't find where it was explicitly set to do so by default, our configuration does not seem to have anything to specifically enable it.

FYI this seems very similar to what i observed with the mysql <-> mw connections https://phabricator.wikimedia.org/T238823#8597868. For that case it seemd like the issue may only be present for cross dc traffic

Adding one more data point that just might be in some cases related: in T292663 we noticed (after going first down the wrong rabbithole) that in some cases we have connections mw (envoy) -> shellbox (envoy) timing out due to the 250ms connect_timeout configuration we have in those envoys. An interesting graph to paste is the p99 of connection establishment timings.

p99 varies a lot and peaks are often above the 200ms mark. Adding also that depooling a host from all traffic made it impossible to reproduce this.

It is not inconceivable some percentage of these RSTs (sent by the client in this case) are because the connection has been already been closed on the client side before a packet, sent by the server, has been received. I don't have data to corroborate this right now, adding it as a hypothesis for when I got time to dive into this deeper.