while running pywikibot on pl.wikipedia I quite often encounter code crashes after page generator encounters 502 Server Error from wikimedia API
What happens?:
script throws exception and crashes
Retrieving 50 pages from wikipedia:pl.
ERROR: Traceback (most recent call last):
File "/home/masti/pw/core/pywikibot/data/api/_requests.py", line 684, in _http_request
response = http.request(self.site, uri=uri,
File "/home/masti/pw/core/pywikibot/comms/http.py", line 283, in request
r = fetch(baseuri, headers=headers, **kwargs)
File "/home/masti/pw/core/pywikibot/comms/http.py", line 457, in fetch
callback(response)
File "/home/masti/pw/core/pywikibot/comms/http.py", line 354, in error_handling_callback
raise ServerError(
pywikibot.exceptions.ServerError: 502 Server Error: Server Hangup
Traceback (most recent call last):
File "/home/masti/pw/core/pwb.py", line 40, in <module>
sys.exit(main())
File "/home/masti/pw/core/pwb.py", line 36, in main
runpy.run_path(str(path), run_name='__main__')
File "/usr/lib/python3.10/runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/masti/pw/core/pywikibot/scripts/wrapper.py", line 521, in <module>
main()
File "/home/masti/pw/core/pywikibot/scripts/wrapper.py", line 505, in main
if not execute():
File "/home/masti/pw/core/pywikibot/scripts/wrapper.py", line 492, in execute
run_python_file(filename, script_args, module)
File "/home/masti/pw/core/pywikibot/scripts/wrapper.py", line 149, in run_python_file
exec(compile(source, filename, 'exec', dont_inherit=True),
File "masti/ms-contains.py", line 433, in <module>
main()
File "masti/ms-contains.py", line 427, in main
bot.run() # guess what it does
File "masti/ms-contains.py", line 147, in run
for page in self.generator:
File "/home/masti/pw/core/pywikibot/pagegenerators/__init__.py", line 660, in PreloadingGenerator
yield from site.preloadpages(group, groupsize=groupsize,
File "/home/masti/pw/core/pywikibot/site/_generators.py", line 202, in preloadpages
for pagedata in rvgen:
File "/usr/lib/python3.10/_collections_abc.py", line 330, in __next__
return self.send(None)
File "/home/masti/pw/core/pywikibot/tools/collections.py", line 279, in send
return next(self._started_gen)
File "/home/masti/pw/core/pywikibot/data/api/_generators.py", line 781, in generator
yield from super().generator
File "/home/masti/pw/core/pywikibot/data/api/_generators.py", line 607, in generator
self.data = self.request.submit()
File "/home/masti/pw/core/pywikibot/data/api/_requests.py", line 993, in submit
response, use_get = self._http_request(use_get, uri, body, headers,
File "/home/masti/pw/core/pywikibot/data/api/_requests.py", line 684, in _http_request
response = http.request(self.site, uri=uri,
File "/home/masti/pw/core/pywikibot/comms/http.py", line 283, in request
r = fetch(baseuri, headers=headers, **kwargs)
File "/home/masti/pw/core/pywikibot/comms/http.py", line 457, in fetch
callback(response)
File "/home/masti/pw/core/pywikibot/comms/http.py", line 354, in error_handling_callback
raise ServerError(
pywikibot.exceptions.ServerError: 502 Server Error: Server Hangup
CRITICAL: Exiting due to uncaught exception ServerError: 502 Server Error: Server HangupWhat should have happened instead?:
wait for server to come back as the errors are temporary
Software version:
Pywikibot: [https] masti01-pywikibot.git (04383ba, g18666, 2024/05/06, 16:04:47, master)
Release version: 9.2.0.dev2
packaging version: 24.0
mwparserfromhell version: 0.6.6
wikitextparser version: n/a
requests version: 2.31.0
cacerts: /home/masti/pw/core/venv/lib/python3.10/site-packages/certifi/cacert.pem
certificate test: ok
Python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]