Overview of Gerrit Session Timeout and Keepalive Settings
Our Gerrit instance has the sshd.idleTimeout set to 3600 seconds (1 hour). This means that if there is no activity over that duration, Gerrit will terminate the session.
While typical git operations are fast and usually do not reach this timeout, we have bots that use gerrit stream-events. These bots will be terminated if no event is generated during inactivity in Gerrit.
TCP Proxy and Bot Configurations
Public bots pass through the TCP Proxy, which currently has a timeout of 3636 seconds. These bots can establish a keepalive (wikibugs2 has one set).
The legacy Zuul does not support keepalive but we have made it to connect directly to the Gerrit server by pointing it to gerrit.discovery.wmnet**. It is still subject to our sshd.idleTimeout = 3600s.
The new Zuul does support keepalive, and I have confirmed it to be keep the connection alive on our development instance in WMCS (which connects through the TCP proxy).
Gerrit default
Gerrit upstream defaults to sshd.idleTimeout = 0 which disables the closing of idle connections and thus it defaults to not terminate gerrit stream-events sessions
Proposed Fixes
The TCP Proxy should have a lower timeout (3600s sounds a lot) and instead the clients should set a keepalive.
For the clients directly connected to Gerrit (TCP proxy or legacy Zuul) they will have a permanently established connection. If they mis disconnect, Gerrit has a TCP keepalive enabled by default [sshd.tcpKeepAlive]. I think that its behavior comes from the underlying Linux Kernel:
/proc/sys/net/ipv4/tcp_keepalive_time:300 /proc/sys/net/ipv4/tcp_keepalive_intvl:1 /proc/sys/net/ipv4/tcp_keepalive_probes:2
Which translates to: after 300 seconds, send a probe every 1 second for up to 2 probe. If there is no response, the kernel closes the connection.
Those TCP Keepalive packets do not carry any payload and my Guess is HAProxy does not take them in account for its timeout. But it is fine if clients will implement SSH level keepalive instead.
Actions
- Lower the TCP proxy idle timeout for port 29418 from 3636 seconds to a lower value (300 seconds?)
- Document the timeout on Wikitech: Gerrit/Bot_account and recommend setting a keepalive when using gerrit stream-events.
- Enable keep alive / switch to gerrit.discovery.wmnet in the Zuul (Continuous-Integration-Infrastructure (Zuul upgrade))
https://gerrit.wikimedia.org/r/c/operations/puppet/+/1269000- @Dzahn found modern Zuul has a 60 seconds keepalive enabled by default <gerrit connection>.keepalive
- Remove sshd.idleTimeout = 3600 seconds** from our config falling back to upstream default of disabling the SSH timeout.