Page MenuHomePhabricator

Gerrit/Jetty flooding logs with "Dispatched Failed!" messages and becoming unresponsive
Closed, ResolvedPublic


Gerrit with the shipped Jetty servlet container causes problems, when
the client connection pool fills up.

Then gerrit's log get spammed with messages like

[2013-02-19 14:55:54,592] WARN : Dispatched Failed! SCEP@44fb799e{l(/<->r(/,d=false,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=1r}-{AsyncHttpConnection@59ed9773,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0} to org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@3699d37e

This spamming renders Gerrit extremely unresponsive. When the problem
hit us around 2013-02-19 12:55:22,464 UTC, spamming through above
messages amounted to about 1Gb of logs in ~2 hours.

The problem is known and documented upstream:

The problem hit us again at 2013-02-21 ~10:50 UTC, and also in one of
the last days.

Possible mitigation strategies are:

  • Trading Jetty for another container (e.g.: Tomcat)

Version: wmf-deployment
Severity: critical



Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 1:41 AM
bzimport added a project: Gerrit.
bzimport set Reference to bz47284.
bzimport added a subscriber: Unknown Object (MLST).

This has been merged & deployed.

We haven't had this problem for the last week, so...knock on wood...I think it's resolved. Going to go ahead and mark this as FIXED.

Will reopen in a heartbeat if we have the problem again.