Page MenuHomePhabricator

windows unicode directory names fail on py2
Closed, DeclinedPublic

Description

Unicode directory names cause problems whether invoked via pwb.py or directly.

C:\pywikibot\Häuser-core>python pwb.py shell
Traceback (most recent call last):
  File "pwb.py", line 117, in <module>
    os.path.join(rewrite_path, 'pywikibot', 'compat'),
  File "C:\Python279\lib\ntpath.py", line 84, in join
    result_path = result_path + p_path
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 12: ordinal not in range(128)
C:\pywikibot\Häuser-core>set PYTHONPATH=.
C:\pywikibot\Häuser-core>python scripts\shell.py
Traceback (most recent call last):
  File "scripts\shell.py", line 44, in <module>
    main(*args)
  File "scripts\shell.py", line 30, in main
    import pywikibot
  File "C:\pywikibot\Hõuser-core\pywikibot\__init__.py", line 32, in <module>
    from pywikibot import config2 as config
  File "C:\pywikibot\Hõuser-core\pywikibot\config2.py", line 307, in <module>
    register_families_folder(os.path.join(os.path.dirname(__file__), 'families')
)
  File "C:\Python279\lib\ntpath.py", line 84, in join
    result_path = result_path + p_path
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 12: ordinal not in range(128)

Even if we can solve those, we then have the problem of paths in the environment:

The solution doesnt look good
https://measureofchaos.wordpress.com/2011/03/04/python-on-windows-unicode-environment-variables/

Event Timeline

jayvdb raised the priority of this task from to Needs Triage.
jayvdb updated the task description. (Show Details)
jayvdb added a project: Pywikibot.
jayvdb added a subscriber: jayvdb.
Restricted Application added subscribers: Aklapper, Unknown Object (MLST). · View Herald TranscriptApr 15 2015, 9:05 AM

Wow… that is a serious shortcoming on Python 2. That you have to patch your way into using Unicode environment :/ I mean supporting putenv() with unicode should simply use the Unicode functionality (as described in the blog you mention). Okay getenv() is a bit more problematic as it can't simply return unicode :(

So I guess we need some logic in tools or so which allows us to set and get unicode environment variables and use that instead of os.environ.

Xqt triaged this task as High priority.Jul 29 2018, 5:03 PM

Decoding sys.argv[0] would solve this issue above
absolute_path = abspath(os.path.dirname(sys.argv[0].decode('utf-8')))
but the same problem occurs when importing:

C:\pwb\GIT\Häuser\core>py -2 pwb.py shell
Traceback (most recent call last):
  File "pwb.py", line 174, in <module>
    import pywikibot
  File "C:\pwb\GIT\Hõuser\core\pywikibot\__init__.py", line 25, in <module>
    from pywikibot.bot import (
  File "C:\pwb\GIT\Hõuser\core\pywikibot\bot.py", line 101, in <module>
    from pywikibot import config2 as config
  File "C:\pwb\GIT\Hõuser\core\pywikibot\config2.py", line 404, in <module>
    register_families_folder(os.path.join(os.path.dirname(__file__), 'families')
)
  File "C:\Python27\lib\ntpath.py", line 85, in join
    result_path = result_path + p_path
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 10: ordinal
 not in range(128)

C:\pwb\GIT\Häuser\core>
Xqt lowered the priority of this task from High to Lowest.Mar 10 2020, 10:50 AM

Declined in favour of Python 3. Python 2 will be dropped at the end of this month.