Page MenuHomePhabricator

http.py - error_handling_callback
Closed, ResolvedPublicBUG REPORT

Description

@PAWS:~$ pwb.py ~/reflinks2.py -start:Центральная_избирательная_комиссия_Республики_Беларусь -v -debug

=== Pywikibot framework v5.0.0 -- Logging header ===
COMMAND: ['/home/paws/reflinks2.py', '-start:Центральная_избирательная_комиссия_Республики_Беларусь', '-v', '-debug']
DATE: 2020-10-20 12:25:26.395092 UTC
VERSION: [https] r-pywikibot-core.git (5ebba28, g13454, 2020/10/19, 16:57:22, n/a)
SYSTEM: posix.uname_result(sysname='Linux', nodename='jupyter--52ubinbot', release='4.19.0-11-amd64', version='#1 SMP Debian 4.19.146-1 (2020-09-17)', machine='x86_64')
CONFIG FILE DIR: /srv/paws
PACKAGES:
  __main__ (/home/paws/reflinks2.py) = ??
  _brotli (/srv/paws/lib/python3.6/site-packages/brotli/_brotli.abi3.so) = ??
  _cffi_backend (/srv/paws/lib/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so) = 1.14.1
  _ctypes (/usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so) = 1.1.0
  _cython_0_29_21 ([path unknown]) = ??
  _decimal (/usr/lib/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so) = 1.70
  blinker (/srv/paws/lib/python3.6/site-packages/blinker/) = 1.4
  brotli (/srv/paws/lib/python3.6/site-packages/brotli/) = ??
  bs4 (/srv/paws/lib/python3.6/site-packages/bs4/) = 4.9.1
  certifi (/srv/paws/lib/python3.6/site-packages/certifi/) = 2020.06.20
  chardet (/srv/paws/lib/python3.6/site-packages/chardet/) = 3.0.4
  cryptography (/srv/paws/lib/python3.6/site-packages/cryptography/) = 3.0
  ctypes (/usr/lib/python3.6/ctypes/) = 1.1.0
  cython_runtime ([path unknown]) = ??
  decimal (/usr/lib/python3.6/decimal.py) = 1.70
  distutils (/usr/lib/python3.6/distutils/) = 3.6.9
  idna (/srv/paws/lib/python3.6/site-packages/idna/) = 2.10
  ipaddress (/usr/lib/python3.6/ipaddress.py) = 1.0
  json (/usr/lib/python3.6/json/) = 2.0.9
  jwt (/srv/paws/lib/python3.6/site-packages/jwt/) = 1.7.1
  logging (/usr/lib/python3.6/logging/) = 0.5.1.2
  lxml (/srv/paws/lib/python3.6/site-packages/lxml/) = 4.5.2
  mpl_toolkits ([path unknown]) = ??
  mwoauth (/srv/paws/lib/python3.6/site-packages/mwoauth/) = 0.3.7
  mwparserfromhell (/srv/paws/lib/python3.6/site-packages/mwparserfromhell/) = 0.5.4
  oauthlib (/srv/paws/lib/python3.6/site-packages/oauthlib/) = 3.1.0
  pkg_resources (/srv/paws/lib/python3.6/site-packages/pkg_resources/) = ??
  platform (/usr/lib/python3.6/platform.py) = 1.0.8
  re (/usr/lib/python3.6/re.py) = 2.2.1
  requests (/srv/paws/lib/python3.6/site-packages/requests/) = 2.24.0
  requests_oauthlib (/srv/paws/lib/python3.6/site-packages/requests_oauthlib/) = 1.3.0
  ruamel ([path unknown]) = ??
  setuptools (/srv/paws/lib/python3.6/site-packages/setuptools/) = 49.2.0
  six (/srv/paws/lib/python3.6/site-packages/six.py) = 1.15.0
  soupsieve (/srv/paws/lib/python3.6/site-packages/soupsieve/) = 2.0.1
  urllib3 (/srv/paws/lib/python3.6/site-packages/urllib3/) = 1.25.10
MODULES:
  2020-10-20 10:20:05 /srv/paws/pwb/setup.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/__metadata__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/_wbtypes.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/bot.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/config2.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/logging.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/_unidata.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/daemonize.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/i18n.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/exceptions.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/plural.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/version.py
  2020-07-30 19:46:59 /srv/paws/pwb/pywikibot/comms/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/comms/http.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/comms/threadedhttp.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/bot_choice.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/_logging.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/formatter.py
  2020-07-30 19:46:59 /srv/paws/pwb/pywikibot/userinterfaces/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/terminal_interface_base.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/transliteration.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/terminal_interface.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/terminal_interface_unix.py
  2020-07-30 19:46:59 /srv/paws/pwb/pywikibot/data/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/data/api.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/login.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/family.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/diff.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/chars.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/site/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/echo.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/site/_decorators.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/site/_siteinfo.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/throttle.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/page/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/textlib.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/pagegenerators.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/date.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/xmlreader.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/proofreadpage.py
  2020-07-30 19:46:59 /srv/paws/pwb/scripts/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/scripts/noreferences.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/families/wikipedia_family.py
=========================================================

=== Pywikibot framework v5.0.0 -- Logging header ===
COMMAND: ['/home/paws/reflinks2.py', '-start:Центральная_избирательная_комиссия_Республики_Беларусь', '-v', '-debug']
DATE: 2020-10-20 12:25:26.656876 UTC
VERSION: [https] r-pywikibot-core.git (5ebba28, g13454, 2020/10/19, 16:57:22, n/a)
SYSTEM: posix.uname_result(sysname='Linux', nodename='jupyter--52ubinbot', release='4.19.0-11-amd64', version='#1 SMP Debian 4.19.146-1 (2020-09-17)', machine='x86_64')
CONFIG FILE DIR: /srv/paws
PACKAGES:
  __main__ (/home/paws/reflinks2.py) = ??
  _brotli (/srv/paws/lib/python3.6/site-packages/brotli/_brotli.abi3.so) = ??
  _cffi_backend (/srv/paws/lib/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so) = 1.14.1
  _ctypes (/usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so) = 1.1.0
  _cython_0_29_21 ([path unknown]) = ??
  _decimal (/usr/lib/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so) = 1.70
  blinker (/srv/paws/lib/python3.6/site-packages/blinker/) = 1.4
  brotli (/srv/paws/lib/python3.6/site-packages/brotli/) = ??
  bs4 (/srv/paws/lib/python3.6/site-packages/bs4/) = 4.9.1
  certifi (/srv/paws/lib/python3.6/site-packages/certifi/) = 2020.06.20
  chardet (/srv/paws/lib/python3.6/site-packages/chardet/) = 3.0.4
  cryptography (/srv/paws/lib/python3.6/site-packages/cryptography/) = 3.0
  ctypes (/usr/lib/python3.6/ctypes/) = 1.1.0
  cython_runtime ([path unknown]) = ??
  decimal (/usr/lib/python3.6/decimal.py) = 1.70
  distutils (/usr/lib/python3.6/distutils/) = 3.6.9
  idna (/srv/paws/lib/python3.6/site-packages/idna/) = 2.10
  ipaddress (/usr/lib/python3.6/ipaddress.py) = 1.0
  json (/usr/lib/python3.6/json/) = 2.0.9
  jwt (/srv/paws/lib/python3.6/site-packages/jwt/) = 1.7.1
  logging (/usr/lib/python3.6/logging/) = 0.5.1.2
  lxml (/srv/paws/lib/python3.6/site-packages/lxml/) = 4.5.2
  mpl_toolkits ([path unknown]) = ??
  mwoauth (/srv/paws/lib/python3.6/site-packages/mwoauth/) = 0.3.7
  mwparserfromhell (/srv/paws/lib/python3.6/site-packages/mwparserfromhell/) = 0.5.4
  oauthlib (/srv/paws/lib/python3.6/site-packages/oauthlib/) = 3.1.0
  pkg_resources (/srv/paws/lib/python3.6/site-packages/pkg_resources/) = ??
  platform (/usr/lib/python3.6/platform.py) = 1.0.8
  re (/usr/lib/python3.6/re.py) = 2.2.1
  requests (/srv/paws/lib/python3.6/site-packages/requests/) = 2.24.0
  requests_oauthlib (/srv/paws/lib/python3.6/site-packages/requests_oauthlib/) = 1.3.0
  ruamel ([path unknown]) = ??
  setuptools (/srv/paws/lib/python3.6/site-packages/setuptools/) = 49.2.0
  six (/srv/paws/lib/python3.6/site-packages/six.py) = 1.15.0
  soupsieve (/srv/paws/lib/python3.6/site-packages/soupsieve/) = 2.0.1
  urllib3 (/srv/paws/lib/python3.6/site-packages/urllib3/) = 1.25.10
MODULES:
  2020-10-20 10:20:05 /srv/paws/pwb/setup.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/__metadata__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/_wbtypes.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/bot.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/config2.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/logging.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/_unidata.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/daemonize.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/i18n.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/exceptions.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/plural.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/version.py
  2020-07-30 19:46:59 /srv/paws/pwb/pywikibot/comms/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/comms/http.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/comms/threadedhttp.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/bot_choice.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/_logging.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/formatter.py
  2020-07-30 19:46:59 /srv/paws/pwb/pywikibot/userinterfaces/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/terminal_interface_base.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/transliteration.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/terminal_interface.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/terminal_interface_unix.py
  2020-07-30 19:46:59 /srv/paws/pwb/pywikibot/data/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/data/api.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/login.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/family.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/diff.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/chars.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/site/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/echo.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/site/_decorators.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/site/_siteinfo.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/throttle.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/page/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/textlib.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/pagegenerators.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/date.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/xmlreader.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/proofreadpage.py
  2020-07-30 19:46:59 /srv/paws/pwb/scripts/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/scripts/noreferences.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/families/wikipedia_family.py
=========================================================
Python 3.6.9 (default, Jul 17 2020, 12:50:27)
[GCC 8.4.0]
LOADING SITE wikipedia:ru VERSION: 1.36.0-wmf.13
Found 1 wikipedia:ru processes running, including this one.
WARNING: /home/paws/reflinks2.py:498: ResourceWarning: unclosed file <_io.BufferedReader name='404-links.txt'>
  listof404pages, 'r', 'latin_1').read()

Retrieving 50 pages from wikipedia:ru.
No changes were needed on [[Центральная избирательная комиссия Республики Беларусь]]
Traceback (most recent call last):
  File "/srv/paws/pwb/pwb.py", line 363, in <module>
    if not main():
  File "/srv/paws/pwb/pwb.py", line 358, in main
    file_package)
  File "/srv/paws/pwb/pwb.py", line 75, in run_python_file
    main_mod.__dict__)
  File "/home/paws/reflinks2.py", line 820, in <module>
    main()
  File "/home/paws/reflinks2.py", line 816, in main
    bot.run()
  File "/home/paws/reflinks2.py", line 542, in run
    ref.url, use_fake_user_agent=self._use_fake_user_agent)
  File "/srv/paws/pwb/pywikibot/tools/__init__.py", line 1450, in wrapper
    return obj(*__args, **__kw)
  File "/srv/paws/pwb/pywikibot/comms/http.py", line 501, in fetch
    request.data = response
  File "/srv/paws/pwb/pywikibot/comms/threadedhttp.py", line 71, in data
    callback(self)
  File "/srv/paws/pwb/pywikibot/comms/http.py", line 349, in error_handling_callback
    raise Server414Error('Too long GET request')
pywikibot.exceptions.Server414Error: Too long GET request
Dropped throttle(s).
Closing network session.
CRITICAL: Exiting due to uncaught exception <class 'pywikibot.exceptions.Server414Error'>
Network session closed.

Event Timeline

Rubin16 created this task.Oct 20 2020, 12:27 PM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptOct 20 2020, 12:27 PM
Rubin16 triaged this task as Lowest priority.Oct 20 2020, 12:27 PM
Mpaa added a subscriber: Mpaa.EditedOct 20 2020, 12:46 PM

Not sure this is a pywikibot error.
Could you find out which is the ref.url it is trying to fetch in:

f = comms.http.fetch(ref.url, use_fake_user_agent=self._use_fake_user_agent)

and post it here?

Not sure this is a pywikibot error.
Could you find out which is the ref.url it is trying to fetch in:

f = comms.http.(ref.url, use_fake_user_agent=self._use_fake_user_agent)

and post it here?

I need some more instruction how I can do it :) sorry for asking

Mpaa added a comment.EditedOct 20 2020, 12:54 PM

Print the url, if possible.
Add

pywikibot.output(ref.url)
f = comms.http.fetch(ref.url, use_fake_user_agent=self._use_fake_user_agent)

I hope you can reproduce the error without going through 100000 links :-)

Mpaa added a comment.Oct 20 2020, 1:07 PM

Or better, add another exception handling after line 620

except pywikibot.Server414Error as e:
    pywikibot.output("Server414Error --> Can't retrieve page {} : {}"
                     .format(ref.url, e))
    continue

and see what is printed when this is triggered.

This comment was removed by Rubin16.
Mpaa added a comment.Oct 20 2020, 2:06 PM

Seems working, strange.
Anyhow I have added Server414Error to the managed exceptions, so the bot won't crash.

Change 635303 had a related patch set uploaded (by Mpaa; owner: Mpaa):
[pywikibot/core@master] [IMPR] add Server414Error in reflink.py

https://gerrit.wikimedia.org/r/635303

Seems working, strange.
Anyhow I have added Server414Error to the managed exceptions, so the bot won't crash.

The patch isn't working for me:

File "/srv/paws/pwb/pywikibot/tools/__init__.py", line 1708, in __getattr__
    return getattr(self._module, attr)
AttributeError: module 'pywikibot' has no attribute 'Server414Error'
Dropped throttle(s).
Closing network session.

To get a link I actually used this option:

pywikibot.output(ref.url)

Change 635303 merged by jenkins-bot:
[pywikibot/core@master] [IMPR] add Server414Error in reflink.py and close file

https://gerrit.wikimedia.org/r/635303

Xqt closed this task as Resolved.Oct 20 2020, 4:18 PM
Xqt claimed this task.

Change 635323 had a related patch set uploaded (by Mpaa; owner: Mpaa):
[pywikibot/core@master] [bugfix] Add Server414Error to pywikibot.init()

https://gerrit.wikimedia.org/r/635323

Mpaa reopened this task as Open.Oct 20 2020, 5:22 PM

Change 635323 merged by jenkins-bot:
[pywikibot/core@master] [bugfix] Add Server414Error to pywikibot.init()

https://gerrit.wikimedia.org/r/635323

Xqt reassigned this task from Xqt to Mpaa.Oct 20 2020, 6:14 PM
Xqt added a subscriber: Xqt.
Mpaa closed this task as Resolved.Oct 20 2020, 7:04 PM