Page MenuHomePhabricator
Create Task
Maniphest T276715

reflinks.py: unknown encoding
Closed, ResolvedPublicBUG REPORT
Actions

Description

== Pywikibot framework v6.0.0.dev0 -- Logging header ===
COMMAND: ['./scripts/reflinks.py', '-always', '-start:Социально-экономический_институт_СГТУ', '-v', 'debug']
DATE: 2021-03-07 16:05:35.276676 UTC
VERSION: [https] r-pywikibot-core (bc28c3b, g14368, 2021/03/07, 13:36:15, n/a)
SYSTEM: posix.uname_result(sysname='Linux', nodename='tools-sgebastion-08', release='4.19.0-0.bpo.14-amd64', version='#1 SMP Debian 4.19.171-2~deb9u1 (2021-02-08)', machine='x86_64')
CONFIG FILE DIR: /mnt/nfs/labstore-secondary-tools-project/rubin16/pywikibot-core

=========================================================
Python 3.5.3 (default, Nov 18 2020, 21:09:16) 
[GCC 6.3.0 20170516]
Found 1 wikipedia:ru processes running, including this one.
Retrieving 50 pages from wikipedia:ru.
WARNING: Http response status 404
HTTP error (404) for http://hakasia.roskazna.ru/page/10586 on [[Социально-экономический институт СГТУ им. Гагарина Ю. А.]]
Dropped throttle(s).

0 pages read
0 pages written
0 pages skipped
Execution time: 3 seconds
Script terminated by exception:

ERROR: LookupError: unknown encoding: Win-1251
Traceback (most recent call last):
  File "pwb.py", line 363, in <module>
    if not main():
  File "pwb.py", line 358, in main
    file_package)
  File "pwb.py", line 75, in run_python_file
    main_mod.__dict__)
  File "./scripts/reflinks.py", line 767, in <module>
    main()
  File "./scripts/reflinks.py", line 763, in main
    bot.run()
  File "/mnt/nfs/labstore-secondary-tools-project/rubin16/pywikibot-core/pywikibot/bot.py", line 1484, in run
    self.treat(page)
  File "./scripts/reflinks.py", line 535, in treat
    ref.url, use_fake_user_agent=self._use_fake_user_agent)
  File "/mnt/nfs/labstore-secondary-tools-project/rubin16/pywikibot-core/pywikibot/tools/__init__.py", line 1478, in wrapper
    return obj(*__args, **__kw)
  File "/mnt/nfs/labstore-secondary-tools-project/rubin16/pywikibot-core/pywikibot/comms/http.py", line 417, in fetch
    response.encoding = _decide_encoding(response, charset)
  File "/mnt/nfs/labstore-secondary-tools-project/rubin16/pywikibot-core/pywikibot/comms/http.py", line 472, in _decide_encoding
    return _try_decode(response.content, header_encoding)
  File "/mnt/nfs/labstore-secondary-tools-project/rubin16/pywikibot-core/pywikibot/comms/http.py", line 456, in _try_decode
    content.decode(encoding)
LookupError: unknown encoding: Win-1251
Dropped throttle(s).
Closing network session.
CRITICAL: Exiting due to uncaught exception <class 'LookupError'>
Network session closed.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
[bugfix] Fix the issue that there are no changes in the log of contributionspywikibot/coremaster+1 -1
[bugfix] check for LookupError exception in _try_decodepywikibot/coremaster+38 -46
Customize query in gerrit

Related Objects

Event Timeline

Rubin16 created this task.Mar 7 2021, 4:07 PM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptMar 7 2021, 4:07 PM
gerritbot added a comment.Mar 7 2021, 4:26 PM

Change 668809 had a related patch set uploaded (by Rubin; owner: Rubin):
[pywikibot/core@master] fixing T276715

https://gerrit.wikimedia.org/r/668809

gerritbot added a project: Patch-For-Review.Mar 7 2021, 4:26 PM
gerritbot added a comment.Mar 7 2021, 4:36 PM

Change 668809 abandoned by Rubin:
[pywikibot/core@master] fixing T276715

Reason:

https://gerrit.wikimedia.org/r/668809

Rubin16 removed a project: Patch-For-Review.Mar 7 2021, 4:37 PM
Rubin16 renamed this task from http.py: unknown encoding: Win-1251 to reflinks.py: unknown encoding.Mar 7 2021, 4:44 PM

not only win-1251.

Page [[Список кораблей Военно-морского флота Российской Федерации]] saved

2473 pages read
13 pages written
0 pages skipped
Execution time: 366 seconds
Read operation time: 0.1 seconds
Write operation time: 28.2 seconds
Script terminated by exception:

ERROR: LookupError: unknown encoding: binary
Traceback (most recent call last):
  File "pwb.py", line 363, in <module>
    if not main():
  File "pwb.py", line 358, in main
    file_package)
  File "pwb.py", line 75, in run_python_file
    main_mod.__dict__)
  File "./scripts/reflinks.py", line 767, in <module>
    main()
  File "./scripts/reflinks.py", line 763, in main
    bot.run()
  File "/mnt/nfs/labstore-secondary-tools-project/rubin16/pywikibot-core/pywikibot/bot.py", line 1484, in run
    self.treat(page)
  File "./scripts/reflinks.py", line 535, in treat
    ref.url, use_fake_user_agent=self._use_fake_user_agent)
  File "/mnt/nfs/labstore-secondary-tools-project/rubin16/pywikibot-core/pywikibot/tools/__init__.py", line 1478, in wrapper
    return obj(*__args, **__kw)
  File "/mnt/nfs/labstore-secondary-tools-project/rubin16/pywikibot-core/pywikibot/comms/http.py", line 417, in fetch
    response.encoding = _decide_encoding(response, charset)
  File "/mnt/nfs/labstore-secondary-tools-project/rubin16/pywikibot-core/pywikibot/comms/http.py", line 472, in _decide_encoding
    return _try_decode(response.content, header_encoding)
  File "/mnt/nfs/labstore-secondary-tools-project/rubin16/pywikibot-core/pywikibot/comms/http.py", line 456, in _try_decode
    content.decode(encoding)
LookupError: unknown encoding: binary
CRITICAL: Exiting due to uncaught except
Xqt triaged this task as High priority.Mar 7 2021, 5:02 PM
Xqt added a subscriber: Mpaa.
Xqt added a subscriber: valhallasw.Mar 7 2021, 5:12 PM
Xqt subscribed.Mar 8 2021, 6:17 AM

Win-1251 is not a valid alias of Python Standard Encodings. Must be windows-1521.

Xqt claimed this task.Mar 8 2021, 8:03 AM
gerritbot added a comment.Mar 8 2021, 8:12 AM

Change 669725 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [bugfix] check for LookupError exception in _try_decode

https://gerrit.wikimedia.org/r/669725

gerritbot added a project: Patch-For-Review.Mar 8 2021, 8:12 AM
Xqt added a comment.Mar 8 2021, 9:08 AM

@Rubin16: Are you able to test this patch above?

Rubin16 added a comment.Mar 8 2021, 10:02 AM
This comment was removed by Rubin16.
Rubin16 added a comment.Mar 8 2021, 11:03 AM

it stopped crashing but it stopped editing too... it is hard to explain the problem, as I can't see the pages where problem happens.
I just see this

Retrieving 50 pages from wikipedia:ru.
Adding references section before Ссылки section...

So, I suppose some page should have been edited now, but there are no changes in the log of contributions: I ran bot for 10-15 minutes and it certainly should have edited some pages.

I've also noticed some new errors in trace:

Can't retrieve page http://www.vpc.org/studies/amroul2006.pdf : HTTPSConnectionPool(host='www.vpc.org', port=443): Max retries exceeded with url: /studies/amroul2006.pdf (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
WARNING: /mnt/nfs/labstore-secondary-tools-project/rubin16/pywikibot-core/pywikibot/data/api.py:2014: ResourceWarning: unclosed <socket.socket fd=7, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.16.3.190', 59808), raddr=('72.10.32.113', 443)>
  uniquedescr, self._data, self._cachetime = pickle.load(f)
WARNING: /usr/lib/python3.5/socket.py:647: ResourceWarning: unclosed <socket.socket fd=14, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.16.3.190', 33590), raddr=('128.65.195.88', 443)>
  self._sock = None
gerritbot added a comment.Mar 8 2021, 2:47 PM

Change 669725 merged by jenkins-bot:
[pywikibot/core@master] [bugfix] check for LookupError exception in _try_decode

https://gerrit.wikimedia.org/r/669725

Xqt mentioned this in rPWBC025f74c948ee: [bugfix] check for LookupError exception in _try_decode.Mar 8 2021, 3:21 PM
gerritbot added a comment.Mar 8 2021, 5:42 PM

Change 669918 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [bugfix] Fix the issue that there are no changes in the log of contributions

https://gerrit.wikimedia.org/r/669918

Xqt added a comment.Mar 8 2021, 5:43 PM

I suppose some page should have been edited now, but there are no changes in the log of contributions

Should be solved with https://gerrit.wikimedia.org/r/669918

gerritbot added a comment.Mar 8 2021, 5:46 PM

Change 669918 merged by jenkins-bot:
[pywikibot/core@master] [bugfix] Fix the issue that there are no changes in the log of contributions

https://gerrit.wikimedia.org/r/669918

Xqt mentioned this in rPWBC1a11f789a979: [bugfix] Fix the issue that there are no changes in the log of contributions.Mar 8 2021, 6:21 PM
Xqt removed a project: Patch-For-Review.Mar 8 2021, 7:07 PM
Xqt closed this task as Resolved.Mar 24 2021, 2:02 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits