Page MenuHomePhabricator
Create Task
Maniphest T266000

http.py - error_handling_callback
Closed, ResolvedPublicBUG REPORT
Actions

Description

@PAWS:~$ pwb.py ~/reflinks2.py -start:Центральная_избирательная_комиссия_Республики_Беларусь -v -debug

=== Pywikibot framework v5.0.0 -- Logging header ===
COMMAND: ['/home/paws/reflinks2.py', '-start:Центральная_избирательная_комиссия_Республики_Беларусь', '-v', '-debug']
DATE: 2020-10-20 12:25:26.395092 UTC
VERSION: [https] r-pywikibot-core.git (5ebba28, g13454, 2020/10/19, 16:57:22, n/a)
SYSTEM: posix.uname_result(sysname='Linux', nodename='jupyter--52ubinbot', release='4.19.0-11-amd64', version='#1 SMP Debian 4.19.146-1 (2020-09-17)', machine='x86_64')
CONFIG FILE DIR: /srv/paws
PACKAGES:
  __main__ (/home/paws/reflinks2.py) = ??
  _brotli (/srv/paws/lib/python3.6/site-packages/brotli/_brotli.abi3.so) = ??
  _cffi_backend (/srv/paws/lib/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so) = 1.14.1
  _ctypes (/usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so) = 1.1.0
  _cython_0_29_21 ([path unknown]) = ??
  _decimal (/usr/lib/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so) = 1.70
  blinker (/srv/paws/lib/python3.6/site-packages/blinker/) = 1.4
  brotli (/srv/paws/lib/python3.6/site-packages/brotli/) = ??
  bs4 (/srv/paws/lib/python3.6/site-packages/bs4/) = 4.9.1
  certifi (/srv/paws/lib/python3.6/site-packages/certifi/) = 2020.06.20
  chardet (/srv/paws/lib/python3.6/site-packages/chardet/) = 3.0.4
  cryptography (/srv/paws/lib/python3.6/site-packages/cryptography/) = 3.0
  ctypes (/usr/lib/python3.6/ctypes/) = 1.1.0
  cython_runtime ([path unknown]) = ??
  decimal (/usr/lib/python3.6/decimal.py) = 1.70
  distutils (/usr/lib/python3.6/distutils/) = 3.6.9
  idna (/srv/paws/lib/python3.6/site-packages/idna/) = 2.10
  ipaddress (/usr/lib/python3.6/ipaddress.py) = 1.0
  json (/usr/lib/python3.6/json/) = 2.0.9
  jwt (/srv/paws/lib/python3.6/site-packages/jwt/) = 1.7.1
  logging (/usr/lib/python3.6/logging/) = 0.5.1.2
  lxml (/srv/paws/lib/python3.6/site-packages/lxml/) = 4.5.2
  mpl_toolkits ([path unknown]) = ??
  mwoauth (/srv/paws/lib/python3.6/site-packages/mwoauth/) = 0.3.7
  mwparserfromhell (/srv/paws/lib/python3.6/site-packages/mwparserfromhell/) = 0.5.4
  oauthlib (/srv/paws/lib/python3.6/site-packages/oauthlib/) = 3.1.0
  pkg_resources (/srv/paws/lib/python3.6/site-packages/pkg_resources/) = ??
  platform (/usr/lib/python3.6/platform.py) = 1.0.8
  re (/usr/lib/python3.6/re.py) = 2.2.1
  requests (/srv/paws/lib/python3.6/site-packages/requests/) = 2.24.0
  requests_oauthlib (/srv/paws/lib/python3.6/site-packages/requests_oauthlib/) = 1.3.0
  ruamel ([path unknown]) = ??
  setuptools (/srv/paws/lib/python3.6/site-packages/setuptools/) = 49.2.0
  six (/srv/paws/lib/python3.6/site-packages/six.py) = 1.15.0
  soupsieve (/srv/paws/lib/python3.6/site-packages/soupsieve/) = 2.0.1
  urllib3 (/srv/paws/lib/python3.6/site-packages/urllib3/) = 1.25.10
MODULES:
  2020-10-20 10:20:05 /srv/paws/pwb/setup.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/__metadata__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/_wbtypes.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/bot.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/config2.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/logging.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/_unidata.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/daemonize.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/i18n.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/exceptions.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/plural.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/version.py
  2020-07-30 19:46:59 /srv/paws/pwb/pywikibot/comms/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/comms/http.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/comms/threadedhttp.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/bot_choice.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/_logging.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/formatter.py
  2020-07-30 19:46:59 /srv/paws/pwb/pywikibot/userinterfaces/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/terminal_interface_base.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/transliteration.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/terminal_interface.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/terminal_interface_unix.py
  2020-07-30 19:46:59 /srv/paws/pwb/pywikibot/data/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/data/api.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/login.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/family.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/diff.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/chars.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/site/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/echo.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/site/_decorators.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/site/_siteinfo.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/throttle.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/page/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/textlib.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/pagegenerators.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/date.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/xmlreader.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/proofreadpage.py
  2020-07-30 19:46:59 /srv/paws/pwb/scripts/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/scripts/noreferences.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/families/wikipedia_family.py
=========================================================

=== Pywikibot framework v5.0.0 -- Logging header ===
COMMAND: ['/home/paws/reflinks2.py', '-start:Центральная_избирательная_комиссия_Республики_Беларусь', '-v', '-debug']
DATE: 2020-10-20 12:25:26.656876 UTC
VERSION: [https] r-pywikibot-core.git (5ebba28, g13454, 2020/10/19, 16:57:22, n/a)
SYSTEM: posix.uname_result(sysname='Linux', nodename='jupyter--52ubinbot', release='4.19.0-11-amd64', version='#1 SMP Debian 4.19.146-1 (2020-09-17)', machine='x86_64')
CONFIG FILE DIR: /srv/paws
PACKAGES:
  __main__ (/home/paws/reflinks2.py) = ??
  _brotli (/srv/paws/lib/python3.6/site-packages/brotli/_brotli.abi3.so) = ??
  _cffi_backend (/srv/paws/lib/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so) = 1.14.1
  _ctypes (/usr/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so) = 1.1.0
  _cython_0_29_21 ([path unknown]) = ??
  _decimal (/usr/lib/python3.6/lib-dynload/_decimal.cpython-36m-x86_64-linux-gnu.so) = 1.70
  blinker (/srv/paws/lib/python3.6/site-packages/blinker/) = 1.4
  brotli (/srv/paws/lib/python3.6/site-packages/brotli/) = ??
  bs4 (/srv/paws/lib/python3.6/site-packages/bs4/) = 4.9.1
  certifi (/srv/paws/lib/python3.6/site-packages/certifi/) = 2020.06.20
  chardet (/srv/paws/lib/python3.6/site-packages/chardet/) = 3.0.4
  cryptography (/srv/paws/lib/python3.6/site-packages/cryptography/) = 3.0
  ctypes (/usr/lib/python3.6/ctypes/) = 1.1.0
  cython_runtime ([path unknown]) = ??
  decimal (/usr/lib/python3.6/decimal.py) = 1.70
  distutils (/usr/lib/python3.6/distutils/) = 3.6.9
  idna (/srv/paws/lib/python3.6/site-packages/idna/) = 2.10
  ipaddress (/usr/lib/python3.6/ipaddress.py) = 1.0
  json (/usr/lib/python3.6/json/) = 2.0.9
  jwt (/srv/paws/lib/python3.6/site-packages/jwt/) = 1.7.1
  logging (/usr/lib/python3.6/logging/) = 0.5.1.2
  lxml (/srv/paws/lib/python3.6/site-packages/lxml/) = 4.5.2
  mpl_toolkits ([path unknown]) = ??
  mwoauth (/srv/paws/lib/python3.6/site-packages/mwoauth/) = 0.3.7
  mwparserfromhell (/srv/paws/lib/python3.6/site-packages/mwparserfromhell/) = 0.5.4
  oauthlib (/srv/paws/lib/python3.6/site-packages/oauthlib/) = 3.1.0
  pkg_resources (/srv/paws/lib/python3.6/site-packages/pkg_resources/) = ??
  platform (/usr/lib/python3.6/platform.py) = 1.0.8
  re (/usr/lib/python3.6/re.py) = 2.2.1
  requests (/srv/paws/lib/python3.6/site-packages/requests/) = 2.24.0
  requests_oauthlib (/srv/paws/lib/python3.6/site-packages/requests_oauthlib/) = 1.3.0
  ruamel ([path unknown]) = ??
  setuptools (/srv/paws/lib/python3.6/site-packages/setuptools/) = 49.2.0
  six (/srv/paws/lib/python3.6/site-packages/six.py) = 1.15.0
  soupsieve (/srv/paws/lib/python3.6/site-packages/soupsieve/) = 2.0.1
  urllib3 (/srv/paws/lib/python3.6/site-packages/urllib3/) = 1.25.10
MODULES:
  2020-10-20 10:20:05 /srv/paws/pwb/setup.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/__metadata__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/_wbtypes.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/bot.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/config2.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/logging.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/_unidata.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/daemonize.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/i18n.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/exceptions.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/plural.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/version.py
  2020-07-30 19:46:59 /srv/paws/pwb/pywikibot/comms/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/comms/http.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/comms/threadedhttp.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/bot_choice.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/_logging.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/formatter.py
  2020-07-30 19:46:59 /srv/paws/pwb/pywikibot/userinterfaces/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/terminal_interface_base.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/transliteration.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/terminal_interface.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/userinterfaces/terminal_interface_unix.py
  2020-07-30 19:46:59 /srv/paws/pwb/pywikibot/data/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/data/api.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/login.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/family.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/diff.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/tools/chars.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/site/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/echo.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/site/_decorators.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/site/_siteinfo.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/throttle.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/page/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/textlib.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/pagegenerators.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/date.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/xmlreader.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/proofreadpage.py
  2020-07-30 19:46:59 /srv/paws/pwb/scripts/__init__.py
  2020-10-20 10:20:05 /srv/paws/pwb/scripts/noreferences.py
  2020-10-20 10:20:05 /srv/paws/pwb/pywikibot/families/wikipedia_family.py
=========================================================
Python 3.6.9 (default, Jul 17 2020, 12:50:27)
[GCC 8.4.0]
LOADING SITE wikipedia:ru VERSION: 1.36.0-wmf.13
Found 1 wikipedia:ru processes running, including this one.
WARNING: /home/paws/reflinks2.py:498: ResourceWarning: unclosed file <_io.BufferedReader name='404-links.txt'>
  listof404pages, 'r', 'latin_1').read()

Retrieving 50 pages from wikipedia:ru.
No changes were needed on [[Центральная избирательная комиссия Республики Беларусь]]
Traceback (most recent call last):
  File "/srv/paws/pwb/pwb.py", line 363, in <module>
    if not main():
  File "/srv/paws/pwb/pwb.py", line 358, in main
    file_package)
  File "/srv/paws/pwb/pwb.py", line 75, in run_python_file
    main_mod.__dict__)
  File "/home/paws/reflinks2.py", line 820, in <module>
    main()
  File "/home/paws/reflinks2.py", line 816, in main
    bot.run()
  File "/home/paws/reflinks2.py", line 542, in run
    ref.url, use_fake_user_agent=self._use_fake_user_agent)
  File "/srv/paws/pwb/pywikibot/tools/__init__.py", line 1450, in wrapper
    return obj(*__args, **__kw)
  File "/srv/paws/pwb/pywikibot/comms/http.py", line 501, in fetch
    request.data = response
  File "/srv/paws/pwb/pywikibot/comms/threadedhttp.py", line 71, in data
    callback(self)
  File "/srv/paws/pwb/pywikibot/comms/http.py", line 349, in error_handling_callback
    raise Server414Error('Too long GET request')
pywikibot.exceptions.Server414Error: Too long GET request
Dropped throttle(s).
Closing network session.
CRITICAL: Exiting due to uncaught exception <class 'pywikibot.exceptions.Server414Error'>
Network session closed.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
[bugfix] Add Server414Error to pywikibot.__init__()pywikibot/coremaster+3 -2
[IMPR] add Server414Error in reflink.py and close filepywikibot/coremaster+3 -2
Customize query in gerrit

Related Objects

Event Timeline

Rubin16 created this task.Oct 20 2020, 12:27 PM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptOct 20 2020, 12:27 PM
Rubin16 triaged this task as Lowest priority.Oct 20 2020, 12:27 PM
Mpaa subscribed.EditedOct 20 2020, 12:46 PM

Not sure this is a pywikibot error.
Could you find out which is the ref.url it is trying to fetch in:

f = comms.http.fetch(ref.url, use_fake_user_agent=self._use_fake_user_agent)

and post it here?

Rubin16 added a comment.Oct 20 2020, 12:50 PM
In T266000#6563997, @Mpaa wrote:

Not sure this is a pywikibot error.
Could you find out which is the ref.url it is trying to fetch in:

f = comms.http.(ref.url, use_fake_user_agent=self._use_fake_user_agent)

and post it here?

I need some more instruction how I can do it :) sorry for asking

Mpaa added a comment.EditedOct 20 2020, 12:54 PM

Print the url, if possible.
Add

pywikibot.output(ref.url)
f = comms.http.fetch(ref.url, use_fake_user_agent=self._use_fake_user_agent)

I hope you can reproduce the error without going through 100000 links :-)

Mpaa added a comment.Oct 20 2020, 1:07 PM

Or better, add another exception handling after line 620

except pywikibot.Server414Error as e:
    pywikibot.output("Server414Error --> Can't retrieve page {} : {}"
                     .format(ref.url, e))
    continue

and see what is printed when this is triggered.

Rubin16 added a comment.Oct 20 2020, 1:36 PM
This comment was removed by Rubin16.
Rubin16 added a comment.Oct 20 2020, 1:48 PM

I think I got it, the link was:

https://365info.kz/2017/03/beriku-imashevu-uporno-ne-dayut-boltatsya-bez-dela-ashimbaev

Mpaa added a comment.Oct 20 2020, 2:06 PM

Seems working, strange.
Anyhow I have added Server414Error to the managed exceptions, so the bot won't crash.

gerritbot added a comment.Oct 20 2020, 2:06 PM

Change 635303 had a related patch set uploaded (by Mpaa; owner: Mpaa):
[pywikibot/core@master] [IMPR] add Server414Error in reflink.py

https://gerrit.wikimedia.org/r/635303

gerritbot added a project: Patch-For-Review.Oct 20 2020, 2:06 PM
Rubin16 added a comment.Oct 20 2020, 2:11 PM
In T266000#6564377, @Mpaa wrote:

Seems working, strange.
Anyhow I have added Server414Error to the managed exceptions, so the bot won't crash.

The patch isn't working for me:

File "/srv/paws/pwb/pywikibot/tools/__init__.py", line 1708, in __getattr__
    return getattr(self._module, attr)
AttributeError: module 'pywikibot' has no attribute 'Server414Error'
Dropped throttle(s).
Closing network session.

To get a link I actually used this option:

pywikibot.output(ref.url)
gerritbot added a comment.Oct 20 2020, 4:14 PM

Change 635303 merged by jenkins-bot:
[pywikibot/core@master] [IMPR] add Server414Error in reflink.py and close file

https://gerrit.wikimedia.org/r/635303

Mpaa mentioned this in rPWBC488550dd25a9: [IMPR] add Server414Error in reflink.py and close file.Oct 20 2020, 4:15 PM
Xqt closed this task as Resolved.Oct 20 2020, 4:18 PM
Xqt claimed this task.
Maintenance_bot removed a project: Patch-For-Review.Oct 20 2020, 5:10 PM
gerritbot added a comment.Oct 20 2020, 5:19 PM

Change 635323 had a related patch set uploaded (by Mpaa; owner: Mpaa):
[pywikibot/core@master] [bugfix] Add Server414Error to pywikibot.init()

https://gerrit.wikimedia.org/r/635323

gerritbot added a project: Patch-For-Review.Oct 20 2020, 5:19 PM
Mpaa reopened this task as Open.Oct 20 2020, 5:22 PM
gerritbot added a comment.Oct 20 2020, 6:05 PM

Change 635323 merged by jenkins-bot:
[pywikibot/core@master] [bugfix] Add Server414Error to pywikibot.init()

https://gerrit.wikimedia.org/r/635323

Mpaa mentioned this in rPWBC7a517b67402b: [bugfix] Add Server414Error to pywikibot.__init__().Oct 20 2020, 6:07 PM
Xqt reassigned this task from Xqt to Mpaa.Oct 20 2020, 6:14 PM
Xqt subscribed.
Mpaa closed this task as Resolved.Oct 20 2020, 7:04 PM
Nintendofan885 removed a project: Patch-For-Review.Oct 21 2020, 7:07 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits