On two occasions, Toolforge Redis stopped accepting new connections with the following error:
Apr 28 13:32:11 tools-redis-7 prometheus-redis-exporter[1095897]: time="2024-04-28T13:32:11Z" level=error msg="Couldn't set client name, err: ERR max number of clients reached"
@taavi suspects this is caused by clients not closing their connections, as Redis by default does not have a connection timeout.
11:32 <taavi> the redis docs say that 'By default recent versions of Redis don't close the connection with the client if the client is idle for many seconds: the connection will remain open forever.', which seems like exactly the sort of thing that would cause this kind of issuue 11:33 <dhinus> yep, did we update the Redis version recently? 11:33 <taavi> not as far as I'm aware 11:33 <dhinus> or maybe some new tool is using the connection without closing it? 11:33 <dhinus> is there a Redis setting to force-close the connections after some time? 11:34 <taavi> yeah, or some network instability causing more connections to drop, or something liek that 11:34 * arturo mumbles a joke about redis wanting to be replaced by valkey 11:34 <dhinus> I will open a task to track this issue 11:34 <taavi> apparently the 'timeout' setting can be used for that 11:34 <taavi> https://redis.io/docs/latest/develop/reference/clients/#client-timeouts