Page MenuHomePhabricator

API QueryGenerator set_query_increment ignored if set_maximum_items is not enabled or module is not limited
Closed, DeclinedPublic

Description

When set_maximum_items is set to -1, as is done to fetch revisions, or in the default None state, the argument given to set_query_increment is ignored.
This means it does not always act like 'step' which is commonly available to the site API , and APISite._generator 'step' argument is put into a set_query_increment call.

This effects the responsiveness of the generators, and other aspects like network & memory utilisation. If the caller wants to process a lot of properties for a large list of items, but prefers to do this in small batches, QueryGenerator will attempt to process all items (it will actually send more items than the API limit for titles, pageids, etc), and will iterate over them all.

QueryGenerator, or a subclass, should send batches of the requested query increment only, and continue sending batches until the requested set of items is processed.

Event Timeline

jayvdb claimed this task.
jayvdb raised the priority of this task from to Needs Triage.
jayvdb updated the task description. (Show Details)
jayvdb added a project: Pywikibot.
jayvdb changed Security from none to None.
jayvdb subscribed.
Xqt removed jayvdb as the assignee of this task.
Xqt subscribed.

-step option could be used in that case