Page MenuHomePhabricator

config2.py UnicodeDecodeError
Closed, ResolvedPublic

Description

Last weekend we tried installing pywikibot on a few windows laptops and we ran into problems with the ones running Windows 10. (Same procedure worked for the Windows 8 laptop).

After downloading and unpacking the zipped file, installing requests, and running python setup.py we tried running python pwb.py version to trigger generating the user_config and got the following error:

NOTE: 'user-config.py' was not found!
Please follow the prompts to create it:
No handlers could be found for logger "pywiki"
Traceback (most recent call last):
  File "pwb.py", line 194, in <module>
    [])
  File "pwb.py", line 120, in run_python_file
    main_mod.__dict__)
  File "generate_user_files.py", line 25, in <module>
    import pywikibot
  File "pywikibot\__init__.py", line 37, in <module>
    from pywikibot import config2 as config
  File "pywikibot\config2.py", line 1059, in <module>
    editor = _detect_win32_editor()
  File "pywikibot\config2.py", line 903, in _detect_win32_editor
    editor = _win32_extension_command(ext)
  File "pywikibot\config2.py", line 890, in _win32_extension_command
    extension, e))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 15: ordinal not in range(128)

The critical line seems to be No handlers could be found for logger "pywiki" as this triggers when trying to run scripts through pwb.py or running generate_user_files.py directly.

It's unclear to me what exactly is causing this and also whether it would always fail or if more instructions must simply be added to the documentation for a Windows 10 install (e.g. you cannot change the character encoding of the command prompt anymore).

Details:
python --version: python 2.7.10
pywikibot version: whichever was in the package/pywikibot/core_stable.zip on 2015-11-29

Details

Related Gerrit Patches:

Event Timeline

Lokal_Profil raised the priority of this task from to Needs Triage.
Lokal_Profil updated the task description. (Show Details)
Lokal_Profil added a project: Pywikibot.

The issue seems to be

  1. no program is configured to open .py files, and
  2. windows generates a non-ascii error message, which we then happily try to decode as ascii

Using Python 3 might help, or alternatively change the line

output('Unable to detect program for file extension "{0}": {1}'.format( extension, e))

to

pass.

Thanks.

I tried a bit more and after a re-boot the above error disappears (but a lot of others appear instead). I removed the pywikibot directory, downloaded 2.0rc3, unpacked, ran python setup.py install and now it works.

Since I didn't do the initial two installs I don't know if my routine differed. I'll try to put my hands on a fresh win 10 laptop to see if there is a step which needs to be added to the windows specific routine to make it work. Best case scenario it is simply that you need to do a re-boot at some point.

XZise added a subscriber: XZise.Dec 8 2015, 11:30 AM

Well 2.0rc3 does not have that error message and thus won't cause that failure.

So it might have been the previous stable release which was installed? When was the stable release last updated?

Also note that Pywikibot Nightlies claims that the "Latest stable version is 2.0rc1", but that is not what you get if you download the file (you get 2.0rc3).

XZise added a subscriber: Ladsgroup.Dec 8 2015, 1:04 PM

No the RCs never had that bit of code which cause the specific error mentioned in the OP. It was added in master some time back but not backported. Regarding the nightlies page, I don't know who can edit that but @Ladsgroup usually does that.

OK thanks for checking.

I'll assume that my instructions weren't followed and they downloaded the wrong pywikibot bundle then (not easily checked now).

I'll leave this open for @Ladsgroup to comment on the nightlies page but it can probably be closed as invalid afterwards.

(alternatively it is an indication that windows 10 won't play nice with the new master but without regular access to a win 10 machine I'm not going to dig deeper into that)

Just got confirmed that at least one of the users used git to get the latest version of core. So whilst my problem is solved (point them to 2.0rc3) there is definitely something going on with master on windows (possibly not only 10).

XZise added a comment.Dec 8 2015, 2:38 PM

Alternatively could you maybe replace the {1} with {1!r} and check if that traceback disappears? It'll probably output the error but it would be interesting to see what the actual error message is.

System error code 2 / The system cannot find the file specified. This is localized, so it's 'Het systeem kan het opgegeven bestand niet vinden' in Dutch and e.g. 'Le système ne trouve pas le fichier spécifié.' in French. On Python 2, that's a str encoded in the current locale (win-1252 for French), which of course does not decode as ascii. On Python 3, it's decoded for us.

jayvdb added a subscriber: jayvdb.Dec 8 2015, 7:51 PM

System error code 2 / The system cannot find the file specified. This is localized, so it's 'Het systeem kan het opgegeven bestand niet vinden' in Dutch and e.g. 'Le système ne trouve pas le fichier spécifié.' in French. On Python 2, that's a str encoded in the current locale (win-1252 for French), which of course does not decode as ascii.

I believe this is caused by Windows locale being quite different. I believe on Windows, some parts of the locale are the user locale on startup, whereas on Unix the locale is C until locale.setlocale(LC_ALL, '') is invoked. Only welcome.py sets the locale to the user locale, but that happens after config2.py is loaded.

My guess is that welcome.py on Linux would also have problems any time it encounters an OSError, which uses strerror.

http://stackoverflow.com/questions/18397572/python-ioerror-strerror-is-not-unicode
https://bugs.launchpad.net/bzr/+bug/273978

I havent found a good recipe for fixing this; my first guess is we should have a wrapper Exception subclass which does the decoding for OSError, and/or add special handling of OSError in pywikibot.logging.exception.

jayvdb renamed this task from Error installing pywikibot on Windows 10 (No handlers could be found for logger "pywiki") to config2.py UnicodeDecodeError on Windows 10.Dec 8 2015, 7:57 PM
jayvdb set Security to None.

The "2.0rc1" is changed to "2.0rc3" now.

Framawiki added a subscriber: Framawiki.EditedApr 17 2017, 8:43 AM

I've the same problem on a VM which runs on a fresh Windows 7 32bits.
As I created it for development, this is the last Git version of pywikibot with Python 2.7.13.

Use {1!r} instead of {1} miraculously solve the problem: now I'm able to use pwb.

Framawiki renamed this task from config2.py UnicodeDecodeError on Windows 10 to config2.py UnicodeDecodeError.Apr 17 2017, 8:50 AM

Change 348444 had a related patch set uploaded (by Gerrit Patch Uploader; owner: Framawiki):
[pywikibot/core@master] config2: Fix UnicodeDecodeError

https://gerrit.wikimedia.org/r/348444

Change 348444 merged by jenkins-bot:
[pywikibot/core@master] config2: use repr() of exceptions to prevent UnicodeDecodeErrors

https://gerrit.wikimedia.org/r/348444

Xqt closed this task as Resolved.Apr 17 2017, 10:39 AM
Xqt claimed this task.
Xqt reassigned this task from Xqt to Framawiki.
Xqt added a subscriber: Xqt.