Page MenuHomePhabricator

Should not die but retry after a cirrussearch-too-busy-error
Closed, ResolvedPublic

Description

With something like

generator = pywikibot.pagegenerators.SearchPageGenerator(i.name, step=50, total=50, site=pywikibot.Site("wikidata", "wikidata"), namespaces=[0])
WARNING: API error cirrussearch-too-busy-error: Search is currently too busy. Please try again later.
API Error: query=
u"{u'inprop': [u'protection'], 'gsrwhat': [u'text'], u'maxlag': ['5'], u'generator': [u'search'], u'format': [u'json'], u'gsrnamespace': [0], 'gsrsearch': [u'Arnaud Cl\\xe9ment'], u'prop': [u'info', u'imageinfo', u'categoryinfo'], u'continue': [True], u'meta': [u'userinfo'], u'indexpageids': [True], u'action': [u'query'], u'gsrlimit': [u'50'], u'iiprop': [u'timestamp', u'user', u'comment', u'url', u'size', u'sha1', u'metadata'], u'uiprop': [u'blockinfo', u'hasmsg']}"
           response=
{u'servedby': u'mw1278', u'error': {u'info': u'Search is currently too busy. Please try again later.', u'code': u'cirrussearch-too-busy-error', u'help': u'See https://www.wikidata.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes.'}}
Traceback (most recent call last):
  File "/shared/pywikipedia/core/pwb.py", line 262, in <module>
    if not main():
  File "/shared/pywikipedia/core/pwb.py", line 255, in main
    run_python_file(filename, [filename] + args, argvu, file_package)
  File "/shared/pywikipedia/core/pwb.py", line 121, in run_python_file
    main_mod.__dict__)
  File "wd.py", line 34, in <module>
    for element in generator:
  File "/mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core/pywikibot/pagegenerators.py", line 2398, in SearchPageGenerator
    for page in site.search(query, total=total, namespaces=namespaces):
  File "/mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core/pywikibot/data/api.py", line 2768, in __iter__
    self.data = self.request.submit()
  File "/mnt/nfs/labstore-secondary-tools-project/pywikibot/public_html/core/pywikibot/data/api.py", line 2204, in submit
    raise APIError(**result['error'])
pywikibot.data.api.APIError: cirrussearch-too-busy-error: Search is currently too busy. Please try again later. [help:See https://www.wikidata.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce&gt; for notice of API deprecations and breaking changes.]
Dropped throttle(s).
CRITICAL: Closing network session.
Network session closed.

Note: I've the same error on the website with my browser. It's fine after a minute.

Event Timeline

Try/except seems doesn't work, my script is always blocked...
Edit: oh, the problem is on the loop, not the generator itself.

Change 365225 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [IMPR] retry after a cirrussearch-too-busy-error

https://gerrit.wikimedia.org/r/365225

Xqt triaged this task as High priority.

@Framawiki: Could you confirm whether it works?

My bot was running during all the night, without you patch, and it didn't encounter this problem. It would appear to be quite rare.
So I can't confirm that you patch will solve this, but the code looks good for me.

Change 365225 merged by jenkins-bot:
[pywikibot/core@master] [IMPR] retry after a cirrussearch-too-busy-error

https://gerrit.wikimedia.org/r/365225