Page MenuHomePhabricator
Create Task
Maniphest T294836

query_limit setting fails with API:Info query
Closed, ResolvedPublicBUG REPORT
Actions

Description

Sample code (simplified):

site = pywikibot.Site('ro', 'wikipedia')
page = pywikibot.Page(site, result['page_title'])
pywikibot.output(page.title())
if page.exists() and page.isRedirectPage():
    page = page.getRedirectTarget()
elif not page.exists():
    pass # DO stuff

Expected outcome: get the redirect target or handle non-existent pages

Actual outcome:

Traceback (most recent call last):
  File "pwb.py", line 420, in <module>
    if not main():
  File "pwb.py", line 415, in main
    module)
  File "pwb.py", line 113, in run_python_file
    main_mod.__dict__)
  File "./wikiro/robots/python/pywikipedia/localitati/create_shortcuts.py", line 80, in <module>
    main()
  File "./wikiro/robots/python/pywikipedia/localitati/create_shortcuts.py", line 64, in main
    if page.exists() and page.isRedirectPage():
  File "/home/andrei/pywikibot-core/pywikibot/page/__init__.py", line 717, in exists
    raise InvalidPageError(self)

Trying to debug, we add a print:

$ git diff pywikibot/page/__init__.py
diff --git a/pywikibot/page/__init__.py b/pywikibot/page/__init__.py
index ee06a0203..fa52faec5 100644
--- a/pywikibot/page/__init__.py
+++ b/pywikibot/page/__init__.py
@@ -714,6 +714,7 @@ class BasePage(ComparableMixin):
         """
         with suppress(AttributeError):
             return self.pageid > 0
+        print(self.pageid)
         raise InvalidPageError(self)

     @property

Sure enough, we had an AttributeError:

Traceback (most recent call last):
  File "pwb.py", line 420, in <module>
    if not main():
  File "pwb.py", line 415, in main
    module)
  File "pwb.py", line 113, in run_python_file
    main_mod.__dict__)
  File "./wikiro/robots/python/pywikipedia/localitati/create_shortcuts.py", line 80, in <module>
    main()
  File "./wikiro/robots/python/pywikipedia/localitati/create_shortcuts.py", line 64, in main
    if source_page.exists():
  File "/home/andrei/pywikibot-core/pywikibot/page/__init__.py", line 717, in exists
    print(self.pageid)
  File "/home/andrei/pywikibot-core/pywikibot/page/__init__.py", line 261, in pageid
    self.site.loadpageinfo(self)
  File "/home/andrei/pywikibot-core/pywikibot/site/_apisite.py", line 1107, in loadpageinfo
    self._update_page(page, query)
  File "/home/andrei/pywikibot-core/pywikibot/site/_apisite.py", line 1081, in _update_page
    for pageitem in query:
  File "/home/andrei/pywikibot-core/pywikibot/data/api.py", line 2740, in __iter__
    yield from super().__iter__()
  File "/home/andrei/pywikibot-core/pywikibot/data/api.py", line 2582, in __iter__
    prev_limit, new_limit, previous_result_had_data)
  File "/home/andrei/pywikibot-core/pywikibot/data/api.py", line 2507, in _handle_query_limit
    self.request[self.prefix + 'limit'] = str(new_limit)
AttributeError: 'PropertyGenerator' object has no attribute 'prefix'
CRITICAL: Exiting due to uncaught exception <class 'AttributeError'>

Pywikibot version:

=== Pywikibot framework v7.0.0.dev0 -- Logging header ===                                                                                     [80/1812]
COMMAND: ['pwb.py', '-v']
DATE: 2021-11-02 13:35:29.761891 UTC
VERSION: [https] r-pywikibot-core (28e31f9, g15398, 2021/09/21, 11:52:45, n/a)
SYSTEM: <anonymized>

Details

SubjectRepoBranchLines +/-
[bugfix] Only handle query limit if module is limitedpywikibot/coremaster+1 -1
Customize query in gerrit

Related Objects

Event Timeline

Strainu created this task.Nov 2 2021, 1:46 PM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptNov 2 2021, 1:46 PM
Strainu updated the task description. (Show Details)Nov 2 2021, 1:53 PM
Xqt subscribed.EditedNov 2 2021, 2:36 PM

The raised exception is not shown in your traceback above. For debugging: which page is it or what is the content of result['page_title']?

Xqt added a subscriber: matej_suchanek.Nov 2 2021, 3:12 PM
Strainu added a comment.Nov 2 2021, 5:06 PM

Pretty much any page, I've encountered it several times. Just checked for [[ro:Cod:SIRUTA:136]], it reproduces

Xqt added a comment.EditedNov 3 2021, 8:18 AM
In T294836#7475355, @Strainu wrote:

Pretty much any page, I've encountered it several times. Just checked for [[ro:Cod:SIRUTA:136]], it reproduces

Can't reproduce it:

>>> import pywikibot
>>> site = pywikibot.Site('ro')
>>> page = pywikibot.Page(site, 'Cod:SIRUTA:136')
>>> page.exists()
True
>>> 
>>> page.isRedirectPage()
True

Does clearing the api cache help?

matej_suchanek added a comment.Nov 3 2021, 3:35 PM

Can't reproduce either. But I see self.prefix is not defined always. I suspect there was some unexpected API response that brought the PropertyGenerator to an inconsistent state. And maybe it's cached somewhere on your side.

Strainu added a comment.EditedNov 3 2021, 5:43 PM

Thank you for your suggestions @Xqt and @matej_suchanek . Unfortunately I am still reproducing the issue after deleting all __pycache__ and apicache-py3 folders, using the simplified example (see below) and commits 28e31f98dcc2b152fde16385172a49f721394ed3 and 758de7a513c061c1741d77b8a684d24bba2c5fd0 .

>>> import pywikibot
>>> site = pywikibot.Site('ro')
>>> page = pywikibot.Page(site, 'Cod:SIRUTA:136')
>>> page.exists()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/andrei/pywikibot-core/pywikibot/page/__init__.py", line 717, in exists
    raise InvalidPageError(self)
pywikibot.exceptions.InvalidPageError: Page [[ro:Cod:SIRUTA:136]] is invalid.
>>> page.isRedirectPage()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/andrei/pywikibot-core/pywikibot/page/__init__.py", line 730, in isRedirectPage
    return self.site.page_isredirect(self)
  File "/home/andrei/pywikibot-core/pywikibot/site/_apisite.py", line 1194, in page_isredirect
    self.loadpageinfo(page)
  File "/home/andrei/pywikibot-core/pywikibot/site/_apisite.py", line 1107, in loadpageinfo
    self._update_page(page, query)
  File "/home/andrei/pywikibot-core/pywikibot/site/_apisite.py", line 1081, in _update_page
    for pageitem in query:
  File "/home/andrei/pywikibot-core/pywikibot/data/api.py", line 2740, in __iter__
    yield from super().__iter__()
  File "/home/andrei/pywikibot-core/pywikibot/data/api.py", line 2582, in __iter__
    prev_limit, new_limit, previous_result_had_data)
  File "/home/andrei/pywikibot-core/pywikibot/data/api.py", line 2507, in _handle_query_limit
    self.request[self.prefix + 'limit'] = str(new_limit)
AttributeError: 'PropertyGenerator' object has no attribute 'prefix'
Xqt added a comment.Nov 4 2021, 9:32 AM

@Strainu: I think your code is mostly equal to release 6.6.1 but I have the same result with it and this code snipped works as expected for me. It even works with 6.6.0, 6.5.0, 6.0.0, 5.0.0. I've no glue what happens here. Either your code is different from framework somehow or the the Python or external packages produced curious results with the api call. What is your version.py result?

Strainu added a comment.Nov 4 2021, 9:46 AM
$ python3 pwb.py version
Pywikibot: [https] r-pywikibot-core (758de7a, g15553, 2021/11/03, 12:54:56, OUTDATED)
Release version: 7.0.0.dev0
setuptools version: 33.1.1
mwparserfromhell version: 0.6.3
wikitextparser version: n/a
requests version: 2.25.1
  cacerts: /home/andrei/.local/lib/python3.5/site-packages/certifi/cacert.pem
    certificate test: ok
Python: 3.5.3 (default, Apr  5 2021, 09:00:41)
[GCC 6.3.0 20170516]
PYWIKIBOT_DIR: Not set
PYWIKIBOT_DIR_PWB:
PYWIKIBOT_NO_USER_CONFIG: Not set
Config base dir: /home/andrei/pywikibot-core
Usernames for family "commons":
        commons: Strainubot
Usernames for family "wikidata":
        wikidata: Strainubot
Usernames for family "wikipedia":
        ro: Strainubot
Strainu added a comment.Nov 4 2021, 9:50 AM
$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

[...]

nothing added to commit but untracked files present (use "git add" to track)
Xqt added a comment.Nov 6 2021, 10:48 AM

Can't find this behaviour with that environment (Python 3.5.3, requests 2.25.1) :-(

Xqt added a comment.EditedNov 6 2021, 1:32 PM

@Strainu: Seems there is an issue with api edit limit restriction with your bot account. Could you please add the following print statements at the first line of _handle_query_limit method in api.py :

print('>>> _handle_query_limit')
print(prev_limit, new_limit, had_data)
print(self.query_limit, self.limit)
from pprint import pprint
pprint(self.__dict__)
print('<<<')

and start the isRedirectPage() test again. I got:

>>> import pywikibot
>>> site = pywikibot.Site('ro')
>>> page = pywikibot.Page(site, 'Cod:SIRUTA:136')
>>> page.isRedirectPage()

>>> _handle_query_limit
None None True
None None
{'_count': 0,
 '_previous_dicts': {},
 '_props': frozenset({'info'}),
 'api_limit': None,
 'continue_name': 'continue',
 'continue_update': <bound method QueryGenerator._continue of <pywikibot.data.api.PropertyGenerator object at 0x000001FAF5891898>>,
 'continuekey': ['info'],
 'limit': None,
 'limited_module': None,
 'modules': ['info'],
 'query_limit': None,
 'request': pywikibot.data.api.Request<wikipedia:ro->'/w/api.php?prop=info&continue=&indexpageids=&action=query&titles=Cod:SIRUTA:136&inprop=protection'>,
 'request_class': <class 'pywikibot.data.api.Request'>,
 'resultkey': 'pages',
 'site': APISite("ro", "wikipedia")}
<<<
True
Strainu added a comment.Nov 6 2021, 4:45 PM
>>> page.isRedirectPage()
>>> _handle_query_limit
None None True
1000 None
{'_count': 0,
 '_previous_dicts': {},
 '_props': frozenset({'info'}),
 'api_limit': 1000,
 'continue_name': 'continue',
 'continue_update': <bound method QueryGenerator._continue of <pywikibot.data.api.PropertyGenerator object at 0x7f529e17ca90>>,
 'continuekey': ['info'],
 'limit': None,
 'limited_module': None,
 'modules': ['info'],
 'query_limit': 1000,
 'request': pywikibot.data.api.Request<wikipedia:ro->'/w/api.php?continue=&action=query&prop=info&titles=Cod:SIRUTA:136&inprop=protection&indexpageids='>,
 'request_class': <class 'pywikibot.data.api.Request'>,
 'resultkey': 'pages',
 'site': APISite("ro", "wikipedia")}
<<<
Strainu added a comment.Nov 6 2021, 4:53 PM

Indeed, removing config.step from user-config.py solved the problem. I don't remember why I added that, maybe it's an inheritance from old versions (I have that config file all the way back from compat branch :D )

Xqt triaged this task as High priority.EditedNov 7 2021, 11:41 AM

config.step is a valid setting but fails here because there is no inlimit with this info query. Thank you for your help to find this bug.

Xqt renamed this task from Calls to pywikibot.Page.exists() throw an InvalidPageError to query_limit setting fails with API:Info query.Nov 7 2021, 11:42 AM
Xqt claimed this task.Nov 7 2021, 12:11 PM
gerritbot added a comment.Nov 7 2021, 12:28 PM

Change 737196 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [bugfix] Only handle query limit if module is limited

https://gerrit.wikimedia.org/r/737196

gerritbot added a project: Patch-For-Review.Nov 7 2021, 12:28 PM
Xqt closed this task as Resolved.Nov 7 2021, 1:02 PM
gerritbot added a comment.Nov 7 2021, 1:12 PM

Change 737196 merged by jenkins-bot:

[pywikibot/core@master] [bugfix] Only handle query limit if module is limited

https://gerrit.wikimedia.org/r/737196

Maintenance_bot removed a project: Patch-For-Review.Nov 7 2021, 2:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits