ConcurrentModificationException on non-grouping query with aggregates in SELECT
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Lucas_Werkmeister_WMDE
	Jul 31 2017, 1:47 PM

Description

The following query (link) results in an exception:

SELECT (COUNT(*) AS ?a) (COUNT(?x) AS ?b) (?b/?a AS ?r) {}

java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
at com.bigdata.rdf.sparql.ast.StaticAnalysis.gatherVarsToMaterialize(StaticAnalysis.java:2058)
at com.bigdata.rdf.sparql.ast.StaticAnalysis.gatherVarsToMaterialize(StaticAnalysis.java:2009)
at com.bigdata.rdf.sparql.ast.eval.AST2BOpUtility.addAggregation(AST2BOpUtility.java:4679)
at com.bigdata.rdf.sparql.ast.AST2BOpUtility.convertQueryBaseWithScopedVars(AST2BOpUtility.java:534)
at com.bigdata.rdf.sparql.ast.eval.AST2BOpUtility.convert(AST2BOpUtility.java:287)
at com.bigdata.rdf.sparql.ast.eval.ASTEvalHelper.optimizeQuery(ASTEvalHelper.java:426)
…

I think the query is legal, according to SPARQL, because GROUP BY is not a requirement for using aggregates (“By default a solution set consists of a single group, containing all solutions.”). If you remove the last projection (?b/?a AS ?r), it works.

Here’s the non-reduced query where I found the error: ratio of mandatory constraints

SELECT (COUNT(*) AS ?total) (COUNT(?status) AS ?mandatory) (?mandatory/?total AS ?ratio) WHERE {
  ?property p:P2302 ?statement.
  OPTIONAL { ?statement pq:P2316 ?status. }
}

Details

	Subject	Repo	Branch	Lines +/-
	Fix Concurrent modification on non-grouping query with aggregates	wikidata/query/blazegraph	master	+102 -17

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Igorkim78	T235759 [TRACKING] WDQS / Blazegraph optimization / bug fixes
		Resolved		debt	T172113 ConcurrentModificationException on non-grouping query with aggregates in SELECT

Event Timeline

Lucas_Werkmeister_WMDE created this task.Jul 31 2017, 1:47 PM

Restricted Application added a project: Discovery-ARCHIVED. · View Herald TranscriptJul 31 2017, 1:47 PM

Restricted Application added subscribers: PokestarFan, Aklapper. · View Herald Transcript

Workaround:

SELECT ?total ?mandatory (?mandatory/?total AS ?ratio) WHERE {
{
SELECT (COUNT(*) AS ?total) (COUNT(?status) AS ?mandatory)  WHERE {
  ?property p:P2302 ?statement.
  OPTIONAL { ?statement pq:P2316 ?status. }
}
}
}

Smalyshev triaged this task as Medium priority.Aug 7 2017, 8:07 PM

Filed https://jira.blazegraph.com/browse/BLZG-9002

Smalyshev moved this task from Incoming to Blazegraph on the Wikidata-Query-Service board.Aug 7 2017, 8:17 PM

If it's the same as on this, I think it broke on February 22.

Note that this can also occur on grouping queries:

SELECT ?x (COUNT(*) AS ?total) (SUM(?y) AS ?ys) (?ys/?total AS ?ratio) WHERE {
  ?x wdt:P31 wd:Q1.
  BIND(1 AS ?y)
}
GROUP BY ?x

And an alternative workaround is to inline the variables into the SELECT, i. e. (SUM(?y)/COUNT(*) AS ?ratio). The advantage of this is that you can still use ?ratio in a HAVING clause on the grouping query (a FILTER in the outer query is less efficient).

Change 533108 had a related patch set uploaded (by Igor Kim; owner: Igor Kim):
[wikidata/query/blazegraph@master] Fix Concurrent modification on non-grouping query with aggregates

https://gerrit.wikimedia.org/r/533108

gerritbot added a project: Patch-For-Review.Aug 28 2019, 10:14 PM

Smalyshev added a project: Discovery-Wikidata-Query-Service-Sprint.Aug 28 2019, 10:38 PM

Smalyshev moved this task from Backlog to Needs review on the Discovery-Wikidata-Query-Service-Sprint board.Aug 28 2019, 11:02 PM

Change 533108 merged by Smalyshev:
[wikidata/query/blazegraph@master] Fix Concurrent modification on non-grouping query with aggregates

https://gerrit.wikimedia.org/r/533108

Maintenance_bot removed a project: Patch-For-Review.Aug 29 2019, 4:12 PM

Smalyshev moved this task from Needs review to Done on the Discovery-Wikidata-Query-Service-Sprint board.Aug 29 2019, 9:38 PM

debt closed this task as Resolved.Sep 5 2019, 6:40 PM

debt claimed this task.

Lucas_Werkmeister_WMDE mentioned this in T235540: StackOverflowError when SPARQL query uses same variable name before and after aggregation.Oct 15 2019, 6:05 PM

Gehel added a parent task: T235759: [TRACKING] WDQS / Blazegraph optimization / bug fixes.Oct 17 2019, 12:43 PM