Page MenuHomePhabricator

interwiki_graph is not working due to an encoding error
Closed, ResolvedPublicBUG REPORT

Description

interwiki_graph fall with

Traceback (most recent call last):
  File "C:\Program Files\Python314\Lib\threading.py", line 1081, in _bootstrap_inner
    self._context.run(self.run)
    ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "C:\Users\DragonMsi\Documents\pywikibot\pywikibot\interwiki_graph.py", line 48, in run
    if self.graph.write(filename, prog='dot', format=fmt):
       ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python314\Lib\site-packages\pydot\core.py", line 1730, in write
    s = self.create(prog, format, encoding=encoding)
  File "C:\Program Files\Python314\Lib\site-packages\pydot\core.py", line 1811, in create
    self.write(fp.name, encoding=encoding)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python314\Lib\site-packages\pydot\core.py", line 1728, in write
    f.write(s)
    ~~~~~~~^^^
  File "C:\Program Files\Python314\Lib\encodings\cp1251.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode characters in position 496-498: character maps to <undefined>

The command line is running in cp866 encoding, so it's not known why Python is trying to use 1251.

When trying to specify the encoding as utf-8, the drop error changes to

File "C:\Program Files\Python314\Lib\site-packages\pydot\core.py", line 1811, in create
    self.write(fp.name, encoding=encoding)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python314\Lib\site-packages\pydot\core.py", line 1728, in write
    f.write(s)
    ~~~~~~~^^^
  File "C:\Program Files\Python314\Lib\encodings\cp1251.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\uade0' in position 190: character maps to <undefined>

Event Timeline

Ivan-r updated the task description. (Show Details)

I changed the pydot core file so that the default encoding is utf-8. After that, everything worked. So need to teach PWB to specify this encoding when transmitting data to this module.

Xqt triaged this task as Medium priority.
Xqt changed the subtype of this task from "Task" to "Bug Report".

Change #1235567 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] IMPR: use explicit utf-8 encoding with GraphSavingThread.graph.write

https://gerrit.wikimedia.org/r/1235567

The command line is running in cp866 encoding, so it's not known why Python is trying to use 1251.

locale.getencoding() is used if encoding is not explicitly given, see https://docs.python.org/3/library/locale.html#locale.getencoding

Manually applied the suggested changes, it seems to be working.

Change #1235567 merged by jenkins-bot:

[pywikibot/core@master] IMPR: use explicit utf-8 encoding with GraphSavingThread.graph.write

https://gerrit.wikimedia.org/r/1235567