Page MenuHomePhabricator

replace.py: Exiting due to uncaught exception <class 'OSError'>
Closed, ResolvedPublic

Description

Command line:

python pwb.py replace -fix:HTML -lang:ar -page:"اللغة البونيقية"

Output:

Retrieving 1 pages from wikipedia:ar.


>>> اللغة البونيقية <<<
@@ -3 +3 @@
- | nativename = {{رمز لغة|xpu|{{خط|Phnx|𐤃𐤁𐤓𐤉𐤌 𐤊𐤍𐤏𐤍𐤉𐤌}}}}<br> [[نقحرة]]: دبريم كنعنيم<br>            نطق:Dabarī+ | nativename = {{رمز لغة|xpu|{{خط|Phnx|𐤃𐤁𐤓𐤉𐤌 𐤊𐤍𐠍
<Unicode console stderr>.write: OSError('WriteConsoleW returned 0, n.value = 0')

0 pages read
0 pages written
0 pages skipped
Execution time: 0 seconds
Script terminated by exception:

ERROR: OSError: WriteConsoleW returned 0, n.value = 0
Traceback (most recent call last):
  File "C:\Users\Mohammed\core\pwb.py", line 364, in <module>
    if not main():
  File "C:\Users\Mohammed\core\pwb.py", line 356, in main
    run_python_file(filename,
  File "C:\Users\Mohammed\core\pwb.py", line 74, in run_python_file
    exec(compile(source, filename, 'exec', dont_inherit=True),
  File ".\scripts\replace.py", line 1098, in <module>
    main()
  File ".\scripts\replace.py", line 1090, in main
    bot.run()
  File "C:\Users\Mohammed\core\pywikibot\bot.py", line 1495, in run
    self.treat(page)
  File ".\scripts\replace.py", line 714, in treat
    pywikibot.showDiff(original_text, new_text, context=context)
  File "C:\Users\Mohammed\core\pywikibot\__init__.py", line 1187, in showDiff
    PatchManager(oldtext, newtext, context=context).print_hunks()
  File "C:\Users\Mohammed\core\pywikibot\diff.py", line 319, in print_hunks
    pywikibot.output('\n'.join(self._generate_diff(super_hunk)
  File "C:\Users\Mohammed\core\pywikibot\logging.py", line 125, in output
    logoutput(text, decoder, newline, INFO, **kwargs)
  File "C:\Users\Mohammed\core\pywikibot\logging.py", line 101, in logoutput
    logger.log(_level, text, extra=context, **kwargs)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1264.0_x64__qbz5n2kfra8p0\lib\logging\__init__.py", line 1512, in log
    self._log(level, msg, args, **kwargs)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1264.0_x64__qbz5n2kfra8p0\lib\logging\__init__.py", line 1589, in _log
    self.handle(record)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1264.0_x64__qbz5n2kfra8p0\lib\logging\__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1264.0_x64__qbz5n2kfra8p0\lib\logging\__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1264.0_x64__qbz5n2kfra8p0\lib\logging\__init__.py", line 952, in handle
    self.emit(record)
  File "C:\Users\Mohammed\core\pywikibot\userinterfaces\terminal_interface_base.py", line 493, in emit
    return self.UI.output(text, targetStream=self.stream)
  File "C:\Users\Mohammed\core\pywikibot\userinterfaces\terminal_interface_base.py", line 237, in output
    self._print(text, targetStream)
  File "C:\Users\Mohammed\core\pywikibot\userinterfaces\terminal_interface_base.py", line 175, in _print
    self._write(text, target_stream)
  File "C:\Users\Mohammed\core\pywikibot\userinterfaces\terminal_interface_base.py", line 136, in _write
    target_stream.write(text)
  File "C:\Users\Mohammed\core\pywikibot\userinterfaces\win32_unicode.py", line 125, in write
    raise IOError('WriteConsoleW returned %r, n.value = %r'
OSError: WriteConsoleW returned 0, n.value = 0
CRITICAL: Exiting due to uncaught exception <class 'OSError'>

Output of version.py:

Pywikibot: [https] r-pywikibot-core (b2dcf1a, g14699, 2021/04/24, 17:37:12, master)
Release version: 6.1.1.dev0
requests version: 2.25.1
    certificate test: ok
Python: 3.9.4 (tags/v3.9.4:1f2e308, Apr  4 2021, 13:27:16) [MSC v.1928 64 bit (AMD64)]

Note:

  • Bot crashes while processing ar:اللغة البونيقية, probably because it contains this text 𐤃𐤁𐤓𐤉𐤌 𐤊𐤍𐤏𐤍𐤉𐤌
  • When viewing the article in my web browser (Google chrome), the text in the article is shown correctly.

Request:

  • Please make the bot skip the page (and not crash) when it encounters such errors. This is to save the time needed to manually restart the bot every time it crashes. Thank you.

Event Timeline

Cannot reproduce it with this command line:

pwb replace -simulate -lang:ar Phnx Phoenix -page:"اللغة البونيقية"

What are your sys Encodings:

>>> import sys
>>> sys.stdin.encoding
'utf-8'
>>> sys.stdout.encoding
'utf-8'
>>> sys.stderr.encoding
'utf-8'
>>>

Cannot reproduce it with this command line:

pwb replace -simulate -lang:ar Phnx Phoenix -page:"اللغة البونيقية"

What are your sys Encodings:

>>> import sys
>>> sys.stdin.encoding
'utf-8'
>>> sys.stdout.encoding
'utf-8'
>>> sys.stderr.encoding
'utf-8'
>>>

It is Windows-1256 under the native Windows cmd.exe

Notes:

  • Just today, I discovered a workaround for this bug, which is to use ConEmu Windows console as described in these steps.
  • Using the ConEmu console, the bot works correctly and edits the page as expected (does not crash).
Xqt triaged this task as Medium priority.Apr 27 2021, 1:37 PM

Notes:

python pwb.py replace -fix:HTML -lang:ar -page:"اللغة البونيقية"

Retrieving 1 pages from wikipedia:ar.


>>> اللغة البونيقية <<<
@@ -3 +3 @@
- | nativename = {{رمز لغة|xpu|{{خط|Phnx|𐤃𐤁𐤓𐤉𐤌 𐤊𐤍𐤏𐤍𐤉𐤌}}}}<br> [[نقحرة]]: دبريم كنعنيم<br>            نطق:Dabarī+ | nativename = {{رمز لغة|xpu|{{خط|Phnx|𐤃𐤁𐤓𐤉𐤌 𐤊𐤍𐤏𐤍          /> [[نقحرة]]: دبريم كنعنيم<br /> نطق:Dabarīm kanaʿnīm

Do you want to accept these changes? ([y]es, [N]o, [e]dit original, edit
[l]atest, open in [b]rowser, [m]ore context, [a]ll, [q]uit): y
Edit summary: بوت: تحويل/تصليح HTML
Page [[اللغة البونيقية]] saved

1 pages read
1 pages written
0 pages skipped
Execution time: 9 seconds
Read operation time: 4.0 seconds
Write operation time: 9.0 seconds
Script terminated successfully.

Output of version.py:

Pywikibot: [https] r-pywikibot-core (5fc9e0f, g16033, 2022/03/01, 23:48:57, master)
Release version: 7.1.0.dev0
setuptools version: 60.9.3
mwparserfromhell version: 0.6.4
wikitextparser version: n/a
requests version: 2.27.1
    certificate test: ok
Python: 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)]

Windows version:

  • Windows 11 version 21H2
  • OS build: 22000.527

Perhaps https://gerrit.wikimedia.org/r/c/pywikibot/core/+/766779 also solves this issue because the underlying problem is solved already in Python 3.6 and this patch is not required (or fails) for newer Pythons

Xqt claimed this task.

Solved with https://gerrit.wikimedia.org/r/c/pywikibot/core/+/766779 because win32_unicode.py is no longer used except with Python 3.5 and will be dropped in Pywikibot 8

Reopened due to T303382

For reference, also, please see T303373

Change 769693 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [cleanup] no longer use win32_unicode for Python 3.6+

https://gerrit.wikimedia.org/r/769693

Change 769693 merged by jenkins-bot:

[pywikibot/core@master] [cleanup] no longer use win32_unicode for Python 3.6+

https://gerrit.wikimedia.org/r/769693