Page MenuHomePhabricator

Fix up Gerrit sshd.idleTimeout
Open, LowPublic

Description

Overview of Gerrit Session Timeout and Keepalive Settings

Our Gerrit instance has the sshd.idleTimeout set to 3600 seconds (1 hour). This means that if there is no activity over that duration, Gerrit will terminate the session.

While typical git operations are fast and usually do not reach this timeout, we have bots that use gerrit stream-events. These bots will be terminated if no event is generated during inactivity in Gerrit.

TCP Proxy and Bot Configurations

Public bots pass through the TCP Proxy, which currently has a timeout of 3636 seconds. These bots can establish a keepalive (wikibugs2 has one set).

The legacy Zuul does not support keepalive but we have made it to connect directly to the Gerrit server by pointing it to gerrit.discovery.wmnet**. It is still subject to our sshd.idleTimeout = 3600s.

The new Zuul does support keepalive, and I have confirmed it to be keep the connection alive on our development instance in WMCS (which connects through the TCP proxy).

Gerrit default

Gerrit upstream defaults to sshd.idleTimeout = 0 which disables the closing of idle connections and thus it defaults to not terminate gerrit stream-events sessions


Proposed Fixes

The TCP Proxy should have a lower timeout (3600s sounds a lot) and instead the clients should set a keepalive.

For the clients directly connected to Gerrit (TCP proxy or legacy Zuul) they will have a permanently established connection. If they mis disconnect, Gerrit has a TCP keepalive enabled by default [sshd.tcpKeepAlive]. I think that its behavior comes from the underlying Linux Kernel:

/proc/sys/net/ipv4/tcp_keepalive_time:300
/proc/sys/net/ipv4/tcp_keepalive_intvl:1
/proc/sys/net/ipv4/tcp_keepalive_probes:2

Which translates to: after 300 seconds, send a probe every 1 second for up to 2 probe. If there is no response, the kernel closes the connection.

Those TCP Keepalive packets do not carry any payload and my Guess is HAProxy does not take them in account for its timeout. But it is fine if clients will implement SSH level keepalive instead.

Actions

Event Timeline

LSobanski triaged this task as Medium priority.Feb 23 2026, 4:51 PM

Change #1266149 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gerrit: update sshd timeouts

https://gerrit.wikimedia.org/r/1266149

ABran-WMF lowered the priority of this task from Medium to Low.

I'm not sure this is still a required, I've updated the timers in 1266149. Please @hashar, let me know if that is no longer required, I'll drop the patch and mark this as Resolved

Change #1266149 abandoned by Arnaudb:

[operations/puppet@production] gerrit: update sshd timeouts

https://gerrit.wikimedia.org/r/1266149

ABran-WMF changed the task status from Open to Stalled.Apr 7 2026, 7:09 AM
hashar changed the task status from Stalled to Open.Apr 8 2026, 2:58 PM

I have filed this task as a follow up to the incident T417497 (CI losts Gerrit events for several days after the switchover behind the CDN). The workaround I found was to bypass the TCP proxy and have Zuul to connect directly to Gerrit.

In the task description I have unchecked the box since none of the timeout have changed. We still have to do it though I did not have quite the time to revisit it since there were other higher priorities.

@ABran-WMF I am not sure why you have abandoned your change https://gerrit.wikimedia.org/r/c/operations/puppet/+/1266149 ? It was two fold:

  1. remove Gerrit sshd.idleTimeout = 3600s which will make Gerrit uses its default of zero and disable the ssh idle timeout. Zuul will then have a permanent connection and will no more reconnect after an hour without events from Gerrit. It is an issue we had for a while but I only discovered it as part of investigating the incident.

The idleTimeout on Gerrit side can certainly be disabled, if something is stall at the TCP level I'll expect the kernel keepalive to eventually get rid of the connection.

  1. the HA Proxy timeout tunnel. That needs a bit more investigation and I don't know much about HAProxy timeout (is that an idle time, max duration of connection?) hence my questions on the change.

And they should be done in independent changes.

Change #1269000 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] zuul: set keepalive for ssh stream event connection

https://gerrit.wikimedia.org/r/1269000

Change #1269000 abandoned by Hashar:

[operations/puppet@production] zuul: set keepalive for ssh stream event connection

Reason:

Modern Zuul Gerrit connection driver has keepalive enabled by default with a 60 seconds interval. Noticed by Daniel in upstream doc: https://zuul-ci.org/docs/zuul/14.0.0/drivers/gerrit.html#attr-%3Cgerrit%20ssh%20connection%3E.keepalive

https://gerrit.wikimedia.org/r/1269000

hashar added a subscriber: Dzahn.