When the Gerrit web service has been switched behind ATS, we witnessed git fetches failing with HTTP 502 errors (T417536).
@Vgutierrez found out that is caused by ATS reusing connections and its timeout of 120 seconds cause it to reuse connections that are are about to be closed by the backend.
ATS connection re-use has been disabled by operations/puppet #1239888, quoting Valentin:
We observed gerrit going away after receiving a request from ATS on certain occasions triggering 502s responses.
This could be related to connection re-use or keep-alive timeout desync.
My interpretation T417536#11622959:
Apache on Gerrit has the default Debian configuration:
# allow persistent connections KeepAlive On # No more than 100 requests for a connection MaxKeepAliveRequests 100 # Wait 5 seconds for another request before closing KeepAliveTimeout 5
Can it be that ATS has a much longer timeout than the Apache 5 seconds timeout? If so i imagine:
- ATS picks up a connection that it considers not having timed out (idling less than whatever timeout it has) but is about to expire on the Apache side (it has been idle for almost 5 seconds).
- ATS sends the headers and at the same time, because the connection has been idling for 5 seconds Apache terminates it (tcp FIN?)
- ATS gets the unexpected termination, gives up and emit a 502 response
Thus if ATS reuses a connection that has been idling for next to 5 seconds on the Gerrit side, it emits a packet there (ATS timeout is 120 s so it considers the connection fine to use) and by the time the packet make it, Gerrit has closed the connection. A FIN is replied, ATS consider the backend to fail and emits a 502.
ATS does retry, but not POST requests and the git fetch do use POST are thus not retries.
When we switched Gerrit from eqiad to codfw, the increased latency between ATS and Gerrit caused the more connections to fall in the gap and the issue happens way more frequently.
Action items
From Valentin:
- Gerrit Apache 2 KeepAliveTimeout needs to be synced with ATS value (120s)
- MaxKeepAliveRequests probably needs to be set to 0
You probably want to do this after moving away from public IPs for gerrit backends or at least after closing port 443 to the Internet