Maniphest T205155

unconnected_pages generator doesn't seem to return all pages
Open, HighPublic
Actions

Assigned To

None

Authored By

	Multichill
	Sep 21 2018, 8:21 PM

Description

One of my bots didn't run for a while so I had to catch up at the unconnected pages (see https://nl.wikipedia.org/w/index.php?title=Speciaal:OngekoppeldePaginas&limit=5000&offset=0&namespace=0 ). I use the -unconnected commandline option (https://phabricator.wikimedia.org/diffusion/PWBC/browse/master/pywikibot/pagegenerators.py$985 ). I noticed that if I run "python pwb.py touch.py -lang:nl -family:wikipedia -namespaces:0 -unconnectedpages" that I don't get all pages.

The generator uses site.unconnected_pages ( https://phabricator.wikimedia.org/diffusion/PWBC/browse/master/pywikibot/site.py$6803 ) which using https://phabricator.wikimedia.org/diffusion/PWBC/browse/master/pywikibot/site.py$1915 gets a api.PageGenerator ( https://phabricator.wikimedia.org/diffusion/PWBC/browse/master/pywikibot/data/api.py$2971 ) which is a subclass of the QueryGenerator ( https://phabricator.wikimedia.org/diffusion/PWBC/browse/master/pywikibot/data/api.py$2568 ).

I think the continue handling is going wrong here. Have a look at https://nl.wikipedia.org/w/api.php?action=query&list=querypage&qppage=UnconnectedPages&format=json&qpoffset=10 . The qpoffset is used for the paging and the continue parameter. I don't think we're using the qpoffset

More info at https://www.mediawiki.org/wiki/API:Querypage

Related Objects

Mentioned In: T207582: templatesWithParams() doesn't seem to get all templates in Hebrew
Mentioned Here: T173293: pagegenerator -unconnectedpages skips items older than 3 years
T196876: tests.site_tests.TestPagePreloading.test_preload_langlinks_count is marked as allowed_failure and should be fixed

Event Timeline

Multichill created this task.Sep 21 2018, 8:21 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 21 2018, 8:21 PM

Multichill updated the task description. (Show Details)Sep 21 2018, 8:26 PM

Mainframe98 added a project: Pywikibot-Wikidata.Sep 21 2018, 8:34 PM

Restricted Application added a subscriber: pywikibot-bugs-list. · View Herald TranscriptSep 21 2018, 8:34 PM

Framawiki added a project: Pywikibot.Sep 23 2018, 9:13 AM

I think the continue handling is going wrong here

That's very likely. I recently tried to fix a continue handling issue in PropertyGenerator, see T196876, but similar issues might also exist in other generators. If so, we should look for a more general solution.

maybe similar with T173293

Xqt triaged this task as High priority.Sep 23 2018, 7:16 PM

If all the generators listed on https://www.mediawiki.org/wiki/API:Querypage follow the same logic in Pywikibot as the unconnected pages one, than probably all of them have the same issue. Maybe make a new subclass QueryPageGenerator that wraps around https://www.mediawiki.org/wiki/API:Querypage ?

Dalba mentioned this in T207582: templatesWithParams() doesn't seem to get all templates in Hebrew.Dec 28 2018, 2:29 AM

qpoffset is used, the issue is same as T173293.

The last request that fetches data is

pywikibot.data.api.Request<wikipedia:nl->'/w/api.php?gqppage=UnconnectedPages&prop=info|imageinfo|categoryinfo&inprop=protection&iiprop=timestamp|user|comment|url|size|sha1|metadata&iilimit=max&generator=querypage&action=query&indexpageids=&continue=gqpoffset||userinfo&gqplimit=500&meta=userinfo&uiprop=blockinfo|hasmsg&maxlag=5&format=json&gqpoffset=10000'>

while

pywikibot.data.api.Request<wikipedia:nl->'/w/api.php?gqppage=UnconnectedPages&prop=info|imageinfo|categoryinfo&inprop=protection&iiprop=timestamp|user|comment|url|size|sha1|metadata&iilimit=max&generator=querypage&action=query&indexpageids=&continue=gqpoffset||userinfo&gqplimit=500&meta=userinfo&uiprop=blockinfo|hasmsg&maxlag=5&format=json&gqpoffset=10500'>

yields

{'batchcomplete': '', 'query': {'querypage': {'name': 'UnconnectedPages'}}, 'limits': {'imageinfo': 500}}

and no more data can be fetched.
I do not know what sets the limit of 10000 in the API.

So you get all the pages with namespace=0 that are in the yielded 10500 pages.

matej_suchanek moved this task from Backlog to Data loading problems on the Pywikibot-Wikidata board.Jul 16 2019, 11:37 AM

Aklapper removed a subscriber: Pywikibot-Wikidata.Mar 22 2020, 9:37 PM

In T205155#4845859, @Mpaa wrote:

I do not know what sets the limit of 10000 in the API.

See https://github.com/wikimedia/mediawiki/blob/0ffef1bb1b3e7de3463552b91549de63a558d4f6/includes/specialpage/QueryPage.php#L693

unconnected_pages generator doesn't seem to return all pagesOpen, HighPublicActions

Description

Related Objects

Event Timeline

unconnected_pages generator doesn't seem to return all pages
Open, HighPublic
Actions