Page MenuHomePhabricator

SMW Ask query offset has a hardcoded limit
Closed, ResolvedPublic

Description

Author: jayhuggins

Description:
SMWQueryProcessor without $params['offset']['upperbound'].

Running an ask query with an offset greater than 5000 returns the results from the query with offset=0 instead. Because of this, the only way to reliably fetch all results for large queries is to set offset=0, limit=<very_large_number> and fetch all results at once. This is less desirable for the user as well as the site host than "paging through" large queries by specifying a limit and incrementing the offset by the limit size for every page queried. But limiting the offset in any way makes this preferred scenario impossible.

The code responsible is the 'upperbound' setting for $params['offset'] in SemanticMediaWiki/includes/SMW_QueryProcessor.php at line 601:

                $params['offset'] = array(
			'type' => 'integer',
			'default' => 0,
			'negatives' => false,
			'upperbound' => 5000 // TODO: make setting
		);

My guess is that the upperbound was intended for $params['limit'], but was added to $params['offset'] by mistake.

The solution is not to make 'upperbound' configurable with a global variable as the comment implies, but simply to remove this line altogether (see attachment).


Version: master
Severity: normal

Attached:

Details

Reference
bz49203

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 1:52 AM
bzimport set Reference to bz49203.
bzimport added a subscriber: Unknown Object (MLST).

I've now hit this issue a few times, but just now did the debugging to figure out what was going on. I've been recently trying Yaron's new Miga [1] project and have been trying to use it with the Wikinosh [2] dataset. Wikinosh has 21,000+ food items and I hit this bug trying to page through results and export everything to Miga [3].

[1] http://migadv.com
[2] http://wikinosh.com/wiki/Main_Page
[3] https://github.com/yaronkoren/miga/issues/4

This bug was fixed with pull request 637 [1]. Thus closing as RESOLVED FIXED.

[1] https://github.com/SemanticMediaWiki/SemanticMediaWiki/pull/637