Page MenuHomePhabricator

Pywikibot does not continue after timeout
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  • run some script on computer which after certain time hibernate.
  • wait until hibernating
  • wake up computer

What happens?:

>>> Ottův slovník naučný/Pelec <<<
ERROR: Traceback (most recent call last):
  File "D:\pwb\pywikibot\data\api\_requests.py", line 684, in _http_request
    response = http.request(self.site, uri=uri,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pwb\pywikibot\comms\http.py", line 283, in request
    r = fetch(baseuri, headers=headers, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pwb\pywikibot\comms\http.py", line 457, in fetch
    callback(response)
  File "D:\pwb\pywikibot\comms\http.py", line 333, in error_handling_callback
    raise ServerError(response)
pywikibot.exceptions.ServerError: HTTPSConnectionPool(host='www.wikidata.org', port=443): Read timed out. (read timeout=45)


5067 read operations
4 write operations
Execution time: 1 hours, 44 minutes, 48 seconds
Read operation time: 1.2 seconds
Write operation time: 1572.0 seconds
Script terminated by exception:

ERROR: HTTPSConnectionPool(host='www.wikidata.org', port=443): Read timed out. (read timeout=45) (ServerError)
Traceback (most recent call last):
  File "D:\pwb\pwb.py", line 40, in <module>
    sys.exit(main())
             ^^^^^^
  File "D:\pwb\pwb.py", line 36, in main
    runpy.run_path(str(path), run_name='__main__')
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "D:\pwb\pywikibot\scripts\wrapper.py", line 521, in <module>
    main()
  File "D:\pwb\pywikibot\scripts\wrapper.py", line 505, in main
    if not execute():
           ^^^^^^^^^
  File "D:\pwb\pywikibot\scripts\wrapper.py", line 492, in execute
    run_python_file(filename, script_args, module)
  File "D:\pwb\pywikibot\scripts\wrapper.py", line 149, in run_python_file
    exec(compile(source, filename, 'exec', dont_inherit=True),
  File "D:\pwb\scripts\ht_s.py", line 592, in <module>
    main()
  File "D:\pwb\scripts\ht_s.py", line 588, in main
    bot.run()
  File "D:\pwb\pywikibot\bot.py", line 1573, in run
    super().run()
  File "D:\pwb\pywikibot\bot.py", line 1488, in run
    self.treat(page)
  File "D:\pwb\pywikibot\bot.py", line 1741, in treat
    self.treat_page()
  File "D:\pwb\pywikibot\bot.py", line 2252, in treat_page
    self.treat_page_and_item(page, item)
  File "D:\pwb\scripts\ht_s.py", line 319, in treat_page_and_item
    self.treat_field(item, page.site, field_item)
  File "D:\pwb\scripts\ht_s.py", line 355, in treat_field
    for target in handler(value, site, item, field):
  File "D:\pwb\scripts\ht_s.py", line 413, in handle_wikibase_item
    linked_item = self.template_link_target(item, site, value)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pwb\scripts\ht_s.py", line 263, in template_link_target
    linked_item = pywikibot.ItemPage.fromPage(linked_page)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pwb\pywikibot\page\_wikibase.py", line 1105, in fromPage
    if not lazy_load and not i.exists():
                             ^^^^^^^^^^
  File "D:\pwb\pywikibot\page\_wikibase.py", line 698, in exists
    self.get(get_redirect=True)
  File "D:\pwb\pywikibot\page\_wikibase.py", line 1166, in get
    data = super().get(force, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pwb\pywikibot\page\_wikibase.py", line 737, in get
    data = WikibaseEntity.get(self, force=force)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pwb\pywikibot\page\_wikibase.py", line 268, in get
    data = self.repo.loadcontent(identification)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pwb\pywikibot\site\_datasite.py", line 203, in loadcontent
    data = req.submit()
           ^^^^^^^^^^^^
  File "D:\pwb\pywikibot\data\api\_requests.py", line 993, in submit
    response, use_get = self._http_request(use_get, uri, body, headers,
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pwb\pywikibot\data\api\_requests.py", line 684, in _http_request
    response = http.request(self.site, uri=uri,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pwb\pywikibot\comms\http.py", line 283, in request
    r = fetch(baseuri, headers=headers, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\pwb\pywikibot\comms\http.py", line 457, in fetch
    callback(response)
  File "D:\pwb\pywikibot\comms\http.py", line 333, in error_handling_callback
    raise ServerError(response)
pywikibot.exceptions.ServerError: HTTPSConnectionPool(host='www.wikidata.org', port=443): Read timed out. (read timeout=45)
CRITICAL: Exiting due to uncaught exception ServerError: HTTPSConnectionPool(host='www.wikidata.org', port=443): Read timed out. (read timeout=45)

What should have happened instead?:

bot should continue in run.

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):
I had version from september 2023, where it works again and again (was possible to have script in run several days in short time windows). Now I updated to version from april 2024 and this issue happen.

Other information (browser name/version, screenshots, etc.):

Event Timeline

@JAnD: What is the command line you are using?
Maybe this issue was introduced upstream with MW 1.43; there are knowns timeouts with wikidata since there, see T359427 for example
and this patch for tests: https://gerrit.wikimedia.org/r/c/pywikibot/core/+/1022548

Try to increase the socket timeout to

socket_timeout = (6.05, 60)

in your user-config.py. Does it help?

I noticed it for claimit.py (cswiki->WD, category with 120k members) and for (modified) harvest_template.py (cswikisource->WD, category with 10 k members)

Change #1029487 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [fix] retry api request on ServerError

https://gerrit.wikimedia.org/r/1029487

Xqt triaged this task as High priority.May 9 2024, 2:51 PM

Change #1029487 merged by jenkins-bot:

[pywikibot/core@master] [fix] retry api request on ServerError

https://gerrit.wikimedia.org/r/1029487