Page MenuHomePhabricator

Generator returns the same pages multiple times each time data is in the requests
Open, MediumPublic

Description

A QueryGenerator might return the same page, because not all data was returned in the first request. When then the second request comes in it contains the same page with additional information. We should only return a single Page object, containing all properties (i.e. from both requests).

I stumbled across this when I was searching for the files with the same hash:

>>> import pywikibot
>>> s = pywikibot.Site('commons', 'commons')
>>> h = '1fd33cb7c6163260585093846174886af2733cd6'
>>> list(s.allimages(sha1=h))
[FilePage(b'File:Star empty.svg'), FilePage(b'File:Star empty.svg')]

Both are about the exact same page but the first request says that there is a continuation which results in the second request which contains the same page.

There is a patch by @Mpaa which promises to fix this and @jayvdb said he has also a patch related to this.

Details

Related Gerrit Patches:

Event Timeline

XZise created this task.Jan 16 2015, 12:59 PM
XZise assigned this task to Mpaa.
XZise raised the priority of this task from to Needs Triage.
XZise updated the task description. (Show Details)
XZise added a project: Pywikibot.
XZise added subscribers: XZise, Mpaa, jayvdb.
Restricted Application added subscribers: Aklapper, Unknown Object (MLST). · View Herald TranscriptJan 16 2015, 12:59 PM
valhallasw triaged this task as Medium priority.Jan 29 2015, 9:30 PM
valhallasw updated the task description. (Show Details)
valhallasw set Security to None.

Change 174827 had a related patch set uploaded (by Mpaa):
[FIX] QueryGenerator: buffer data before yielding

https://gerrit.wikimedia.org/r/174827

Dvorapa added a subscriber: Xqt.May 20 2018, 10:36 AM
D3r1ck01 moved this task from Backlog to Needs Review on the Pywikibot board.Nov 5 2018, 11:37 AM