Page MenuHomePhabricator

cosmetic_changes.py: Exiting due to uncaught exception UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd9 in position 80: invalid continuation byte
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue:

python pwb.py cosmetic_changes -always -page:"نقاش المستخدم:Meno25"

Output:

Retrieving 1 pages from wikipedia:ar.


>>> نقاش المستخدم:Meno25 <<<

1 read operation
Execution time: 2 seconds
Read operation time: 2.0 seconds
Script terminated by exception:

ERROR: 'utf-8' codec can't decode byte 0xd9 in position 80: invalid continuation byte (UnicodeDecodeError)
Traceback (most recent call last):
  File "C:\Users\Mohammed\Downloads\core\pwb.py", line 40, in <module>
    sys.exit(main())
             ~~~~^^
  File "C:\Users\Mohammed\Downloads\core\pwb.py", line 36, in main
    runpy.run_path(str(path), run_name='__main__')
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 287, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Mohammed\Downloads\core\pywikibot\scripts\wrapper.py", line 569, in <module>
    main()
    ~~~~^^
  File "C:\Users\Mohammed\Downloads\core\pywikibot\scripts\wrapper.py", line 552, in main
    if not execute():
           ~~~~~~~^^
  File "C:\Users\Mohammed\Downloads\core\pywikibot\scripts\wrapper.py", line 539, in execute
    run_python_file(filename, script_args, module)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mohammed\Downloads\core\pywikibot\scripts\wrapper.py", line 154, in run_python_file
    exec(compile(source, filename, 'exec', dont_inherit=True),
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
         main_mod.__dict__)
         ^^^^^^^^^^^^^^^^^^
  File "C:\Users\Mohammed\Downloads\core\scripts\cosmetic_changes.py", line 131, in <module>
    main()
    ~~~~^^
  File "C:\Users\Mohammed\Downloads\core\scripts\cosmetic_changes.py", line 127, in main
    bot.run()
    ~~~~~~~^^
  File "C:\Users\Mohammed\Downloads\core\pywikibot\bot.py", line 1595, in run
    self.treat(page)
    ~~~~~~~~~~^^^^^^
  File "C:\Users\Mohammed\Downloads\core\pywikibot\bot.py", line 1844, in treat
    self.treat_page()
    ~~~~~~~~~~~~~~~^^
  File "C:\Users\Mohammed\Downloads\core\scripts\cosmetic_changes.py", line 85, in treat_page
    new_text = cc_toolkit.change(old_text)
  File "C:\Users\Mohammed\Downloads\core\pywikibot\cosmetic_changes.py", line 314, in change
    new_text = self._change(text)
  File "C:\Users\Mohammed\Downloads\core\pywikibot\cosmetic_changes.py", line 308, in _change
    text = self.safe_execute(method, text)
  File "C:\Users\Mohammed\Downloads\core\pywikibot\cosmetic_changes.py", line 294, in safe_execute
    result = method(text)
  File "C:\Users\Mohammed\Downloads\core\pywikibot\cosmetic_changes.py", line 679, in cleanUpLinks
    return textlib.replaceExcept(text, linkR, handleOneLink,
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                 ['comment', 'math', 'nowiki', 'pre',
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                  'startspace'])
                                  ^^^^^^^^^^^^^^
  File "C:\Users\Mohammed\Downloads\core\pywikibot\textlib.py", line 452, in replaceExcept
    replacement = new(match)
  File "C:\Users\Mohammed\Downloads\core\pywikibot\cosmetic_changes.py", line 560, in handleOneLink
    titleWithSection = url2string(match['titleWithSection'],
                                  encodings=self.site.encodings())
  File "C:\Users\Mohammed\Downloads\core\pywikibot\tools\chars.py", line 138, in url2string
    raise cast(BaseException, first_exception)
  File "C:\Users\Mohammed\Downloads\core\pywikibot\tools\chars.py", line 132, in url2string
    return unquote(title, enc, errors='strict')
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.3312.0_x64__qbz5n2kfra8p0\Lib\urllib\parse.py", line 712, in unquote
    return ''.join(_generate_unquoted_parts(string, encoding, errors))
           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.3312.0_x64__qbz5n2kfra8p0\Lib\urllib\parse.py", line 688, in _generate_unquoted_parts
    yield _unquote_impl(ascii_match[1]).decode(encoding, errors)
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd9 in position 80: invalid continuation byte
CRITICAL: Exiting due to uncaught exception UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd9 in position 80: invalid continuation byte

Request:

  • Instead of crashing, the bot should skip such pages (ignore this exception) and continue working on other pages.

Software version:

Pywikibot: [https] r-pywikibot-core (42c13c5, g20335, 2026/04/09, 14:21:29, master)
Release version: 11.2.0.dev0
packaging version: 26.0
mwparserfromhell version: 0.7.2
wikitextparser version: 0.56.4
requests version: 2.33.1
    certificate test: ok
Python: 3.13.12 (tags/v3.13.12:1cbe481, Feb  3 2026, 18:22:25) [MSC v.1944 64 bit (AMD64)]
User-Agent: MenoBot/version (wikipedia:ar; User:MenoBot) Pywikibot/11.2.0.dev0 (g20335) Python/3.13.12.final.0 requests/2.33.1
PYWIKIBOT_DIR: Not set
PYWIKIBOT_DIR_PWB: C:\Users\Mohammed\Downloads\core\pywikibot\scripts
PYWIKIBOT_NO_USER_CONFIG: Not set
Config base dir: C:\Users\Mohammed\Downloads\core

Event Timeline

Change #1275355 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] [bugfix] Ignote UnicodeDecodeError in cosmetic_changes.cleanUpLinks

https://gerrit.wikimedia.org/r/1275355

Xqt changed the task status from Open to In Progress.Apr 20 2026, 9:34 AM
Xqt claimed this task.
Xqt triaged this task as High priority.

Change #1275355 merged by Meno25:

[pywikibot/core@master] [bugfix] Ignote UnicodeDecodeError in cosmetic_changes.cleanUpLinks

https://gerrit.wikimedia.org/r/1275355