Gerrit with the shipped Jetty servlet container causes problems, when
the client connection pool fills up.
Then gerrit's log get spammed with messages like
[2013-02-19 14:55:54,592] WARN org.eclipse.jetty.io.nio : Dispatched Failed! SCEP@44fb799e{l(/127.0.0.1:47357)<->r(/127.0.0.1:8080),d=false,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=1r}-{AsyncHttpConnection@59ed9773,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0} to org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@3699d37e
This spamming renders Gerrit extremely unresponsive. When the problem
hit us around 2013-02-19 12:55:22,464 UTC, spamming through above
messages amounted to about 1Gb of logs in ~2 hours.
The problem is known and documented upstream:
http://code.google.com/p/gerrit/wiki/Scaling#Jetty
The problem hit us again at 2013-02-21 ~10:50 UTC, and also in one of
the last days.
Possible mitigation strategies are:
- hand connection limiting off to apache https://gerrit.wikimedia.org/r/#/c/50591
- Trading Jetty for another container (e.g.: Tomcat)
Version: wmf-deployment
Severity: critical