Page MenuHomePhabricator
Create Task
Maniphest T62940

Gerrit throwing internal server error with "status:closed" queries
Closed, ResolvedPublic
Actions

Description

Something goofy is going on with "status:closed" queries in Gerrit lately.

Works (status:abandoned):

$ curl "https://gerrit.wikimedia.org/r/changes/?q=status:abandoned&n=1"
)]}'
[

{
  "kind": "gerritcodereview#change",
  "id": "mediawiki%2Fextensions%2FFlow~master~Iade500a4c2fb8b96e88e5074c7ceff484f02cce9",
  "project": "mediawiki/extensions/Flow",
  "branch": "master",
  "change_id": "Iade500a4c2fb8b96e88e5074c7ceff484f02cce9",
  "subject": "Implement former SimpleAntiSpam extension in \\Flow\\View",
  "status": "ABANDONED",
  "created": "2014-02-06 04:22:23.000000000",
  "updated": "2014-02-06 04:30:14.000000000",
  "mergeable": true,
  "_sortkey": "002af04e0001b469",
  "_number": 111721,
  "owner": {
    "name": "Legoktm"
  },
  "_more_changes": true
}

]

Fails (status:closed):

$ curl "https://gerrit.wikimedia.org/r/changes/?q=status:closed&n=1"
Internal server error

You can reproduce the same behavior by going to https://gerrit.wikimedia.org and inputting "status:closed" into the search input. Doing so results in:

Code Review - Error

500 Internal server error

[Continue]

This is a breaking change in the Gerrit Web API, causing [[mw:Gerrit/Reports]] to stop updating.

Version: wmf-deployment
Severity: major
URL: https://code.google.com/p/gerrit/issues/detail?id=2392

Details

Reference
bz60940

Event Timeline

• bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:00 AM
• bzimport added projects: Gerrit, Upstream.
• bzimport set Reference to bz60940.
• bzimport added a subscriber: Unknown Object (MLST).
• MZMcBride created this task.Feb 6 2014, 4:47 AM
hashar added a comment.Feb 6 2014, 2:50 PM

Same in the search box (which uses the REST API I guess).

Since status:closes is equivalent to is:merged OR is:abandoned I tried that, and it fails as well.

• demon added a comment.Feb 6 2014, 6:43 PM

Pretty sure this is the exception we're seeing (swap merged for abandoned or closed and you'll find some more):

[2014-02-06 14:50:28,927] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /r/changes/?q=is:merged+OR+is:abandoned&n=25&O=1
java.sql.SQLException: setMaxRows() out of range. 2147483647 > 50000000.

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:988)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:974)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)
at com.mysql.jdbc.StatementImpl.setMaxRows(StatementImpl.java:2745)
at com.google.gerrit.reviewdb.client.Change_Access_changes_GwtOrm$$13.allClosedNext(Unknown Source)
at com.google.gerrit.server.query.change.SqlRewriterImpl$13.scan(SqlRewriterImpl.java:340)
at com.google.gerrit.server.query.change.SqlRewriterImpl$PaginatedSource.scan(SqlRewriterImpl.java:645)
at com.google.gerrit.server.query.change.SqlRewriterImpl$ChangeSource.read(SqlRewriterImpl.java:608)
at com.google.gerrit.server.query.change.OrSource.read(OrSource.java:43)
at com.google.gerrit.server.query.change.AndSource.readImpl(AndSource.java:110)
at com.google.gerrit.server.query.change.AndSource.read(AndSource.java:94)
at com.google.gerrit.server.query.change.QueryProcessor.queryChanges(QueryProcessor.java:261)
at com.google.gerrit.server.query.change.QueryChanges.query0(QueryChanges.java:153)
at com.google.gerrit.server.query.change.QueryChanges.query(QueryChanges.java:141)
at com.google.gerrit.server.query.change.QueryChanges.apply(QueryChanges.java:108)
at com.google.gerrit.server.query.change.QueryChanges.apply(QueryChanges.java:41)
at com.google.gerrit.httpd.restapi.RestApiServlet.service(RestApiServlet.java:306)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:70)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:113)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:68)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129)
at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:365)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:744)
• MZMcBride added a comment.Feb 7 2014, 2:36 AM

Chad or someone else who likely remembers where the Gerrit bug tracker is and has an account over there: can you please check if there's already an upstream ticket filed about this issue and if not, copy and paste this one over there and cross-reference?

Aklapper added a comment.Feb 7 2014, 9:30 AM

(In reply to comment #3)

someone else who likely remembers where the Gerrit bug tracker is

https://www.mediawiki.org/wiki/Bug_management/Upstream_bugtrackers

matmarex added a comment.Feb 7 2014, 12:28 PM

Upstream bug seems to be https://code.google.com/p/gerrit/issues/detail?id=2392

QChris added a comment.Feb 7 2014, 12:49 PM

I can reproduce on a local gerrit that uses MySQL, so it's not limited
to our wmf gerrit.

Changing the database to H2 makes the error disappear.

It seems the issue is limited to the MySQL backend, as the error is
thrown from the MySQL connector, whose hard limit for 50000000 rows is
below 2^31-1.

I checked gerrit upstreams stable-2.8 branch, and it does not seem to
have a fix for the issue. But it might actually be a gwtorm issue,
where the setMaxRows gets called :-(

P.S.: Although the bug is of course valid nonetheless, searches for
"status:closed" have never been working IIRC, as they boil down to
using "OR". Previously they just gave the empty result set more often
than not :-(

QChris added a comment.Feb 7 2014, 12:54 PM

Although it of course sucks, you can work around the issues, by resolving the
"OR" part of the query by hand:

Like instead of

curl "https://gerrit.wikimedia.org/r/changes/?q=status:closed"

do

curl "https://gerrit.wikimedia.org/r/changes/?q=status:merged"
curl "https://gerrit.wikimedia.org/r/changes/?q=status:abandoned"

.

gerritbot added a comment.Feb 8 2014, 11:40 AM

Change 112201 had a related patch set uploaded by QChris:
Keep status:closed's limit below MySQL Connector/J's hard limit

https://gerrit.wikimedia.org/r/112201

gerritbot added a comment.Feb 10 2014, 10:31 PM

Change 112201 merged by QChris:
Keep status:closed's limit below MySQL Connector/J's hard limit

https://gerrit.wikimedia.org/r/112201

QChris added a comment.Feb 11 2014, 10:04 AM

New, fixed gerrit has been deployed yesterday.

matmarex added a comment.Feb 11 2014, 12:18 PM

Thank you!

• MZMcBride added a comment.Feb 12 2014, 4:03 AM

\o/

Krenair moved this task from Backlog to Patch merged upstream on the Upstream board.Mar 29 2015, 9:07 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits