Page MenuHomePhabricator

zuul git-daemon sometime reject connections: Too many children, dropping connection
Closed, ResolvedPublic

Description

The git-daemon process (which serves the zuul-merger git repositories) sometimes gets too many connections. It defaults to 32 however:

$ sudo zgrep 'Too many children, dropping connection' /var/log/git-daemon/syslog.log*
/var/log/git-daemon/syslog.log:Jan 18 22:12:16 contint1001 git-daemon[830]: Too many children, dropping connection
/var/log/git-daemon/syslog.log:Feb  6 18:12:16 contint1001 git-daemon[830]: Too many children, dropping connection
/var/log/git-daemon/syslog.log:Feb  6 18:12:17 contint1001 git-daemon[830]: Too many children, dropping connection
/var/log/git-daemon/syslog.log:Feb  6 18:12:18 contint1001 git-daemon[830]: Too many children, dropping connection
/var/log/git-daemon/syslog.log:Mar 11 04:08:53 contint1001 git-daemon[830]: Too many children, dropping connection
/var/log/git-daemon/syslog.log:Apr  9 00:09:13 contint1001 git-daemon[830]: Too many children, dropping connection
/var/log/git-daemon/syslog.log:Apr  9 00:09:14 contint1001 git-daemon[830]: Too many children, dropping connection
/var/log/git-daemon/syslog.log:Apr 16 16:46:51 contint1001 git-daemon[830]: Too many children, dropping connection

contint2001 does not have any such messages.

the logs span more than a year. So that is fairly recent.

Event Timeline

Change 508408 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] contint: bump git-daemon max connections 32 -> 48

https://gerrit.wikimedia.org/r/508408

Change 508408 merged by Dzahn:
[operations/puppet@production] contint: bump git-daemon max connections 32 -> 48

https://gerrit.wikimedia.org/r/508408

Change 733000 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] zuul: double git-daemon max connections 48 -> 96

https://gerrit.wikimedia.org/r/733000

Change 733000 merged by Dzahn:

[operations/puppet@production] zuul: double git-daemon max connections 48 -> 96

https://gerrit.wikimedia.org/r/733000

After raising to 96 max connections, I have checked on contint2001 and contint1001 with:

sudo zgrep 'Too many children, dropping connection' /var/log/git-daemon/syslog.log*

Last messages were on October 22nd. So the new 96 limit seems appropriate.