Page MenuHomePhabricator

Partial failed import of pywikibot causes unrecoverable state
Closed, ResolvedPublic

Description

After 7634a05 , an interesting problem occurs when using pwb.py without a user-config.py file.

$ python pwb.py generate_family_file 'https://wiki.musicbrainz.org/' musicbrainz
NOTE: 'user-config.py' was not found!
Traceback (most recent call last):
  File "pwb.py", line 239, in <module>
    if not main():
  File "pwb.py", line 233, in main
    run_python_file(filename, [filename] + args, argvu, file_package)
  File "pwb.py", line 111, in run_python_file
    main_mod.__dict__)
  File "./generate_family_file.py", line 36, in <module>
    from pywikibot.site_detect import MWSite as Wiki
  File "./pywikibot/__init__.py", line 41, in <module>
    from pywikibot.exceptions import (
  File "./pywikibot/exceptions.py", line 523, in <module>
    import pywikibot.data.api
  File "./pywikibot/data/api.py", line 36, in <module>
    from pywikibot.comms import http
  File "./pywikibot/comms/http.py", line 72, in <module>
    pywikibot.debug(u"Loading cookies failed.", _logger)
  File "/home/travis/build/wikimedia/pywikibot-core/pywikibot/logging.py", line 170, in debug
    logoutput(text, decoder, newline, DEBUG, layer, **kwargs)
  File "/home/travis/build/wikimedia/pywikibot-core/pywikibot/logging.py", line 80, in logoutput
    _init()
  File "/home/travis/build/wikimedia/pywikibot-core/pywikibot/logging.py", line 38, in _init
    init_routine()
  File "./pywikibot/bot.py", line 275, in init_handlers
    pywikibot.tools.debug = debug
AttributeError: 'module' object has no attribute 'tools'

https://travis-ci.org/wikimedia/pywikibot-core/jobs/81149440

pywikibot.tools.debug = debug is an unnecessary line which should have been removed in d7d7a146 , but this is not relevant. When that line is removed, the failure is instead

$ python pwb.py generate_family_file 'https://wiki.musicbrainz.org/' musicbrainz
NOTE: 'user-config.py' was not found!
Traceback (most recent call last):
  File "pwb.py", line 239, in <module>
    if not main():
  File "pwb.py", line 233, in main
    run_python_file(filename, [filename] + args, argvu, file_package)
  File "pwb.py", line 111, in run_python_file
    main_mod.__dict__)
  File "./generate_family_file.py", line 36, in <module>
    from pywikibot.site_detect import MWSite as Wiki
  File "./pywikibot/__init__.py", line 41, in <module>
    from pywikibot.exceptions import (
  File "./pywikibot/exceptions.py", line 527, in <module>
    @pywikibot.tools.deprecated
AttributeError: 'module' object has no attribute 'tools'

Again, removing use of pywikibot.tools.deprecated doesnt fix the problem. many, many uses of pywikibot.tools need to be removed before the problem goes away.

The simple fix is for pwb.py to remove pywikibot.tools and pywikibot.logging from sys.modules.

When pwb first attempts to import pywikibot without a user-config.py present, pywikibot.config fails causing 'pywikibot' import to fail. However pywikibot.tools and pywikibot.logging will still be in sys.modules. This caused errors when pwb then attempts to import generate_family_file as the pywikibot import sequence is different the second time.

Why is still not known.

Possible fixes include:

Event Timeline

jayvdb raised the priority of this task from to Unbreak Now!.
jayvdb updated the task description. (Show Details)
jayvdb added a project: Pywikibot.
jayvdb subscribed.

Change 239640 had a related patch set uploaded (by John Vandenberg):
Do not import tools in config

https://gerrit.wikimedia.org/r/239640

Change 239645 had a related patch set uploaded (by John Vandenberg):
Remove pywikibot modules if import failed

https://gerrit.wikimedia.org/r/239645

jayvdb set Security to None.

Change 239643 had a related patch set uploaded (by John Vandenberg):
Dont overwrite tools.debug

https://gerrit.wikimedia.org/r/239643

Change 239648 had a related patch set uploaded (by John Vandenberg):
Remove use of pywikibot within init

https://gerrit.wikimedia.org/r/239648

Note that merely moving APIError and UploadWarning into exceptions does not fix the problem. Then it looks like this:

$ python pwb.py generate_family_file 'https://wiki.musicbrainz.org/' musicbrainz
NOTE: 'user-config.py' was not found!
Traceback (most recent call last):
  File "pwb.py", line 239, in <module>
    if not main():
  File "pwb.py", line 233, in main
    run_python_file(filename, [filename] + args, argvu, file_package)
  File "pwb.py", line 111, in run_python_file
    main_mod.__dict__)
  File "./generate_family_file.py", line 36, in <module>
    from pywikibot.site_detect import MWSite as Wiki
  File "./pywikibot/__init__.py", line 41, in <module>
    from pywikibot.exceptions import (
  File "./pywikibot/exceptions.py", line 580, in <module>
    @pywikibot.tools.deprecated
AttributeError: 'module' object has no attribute 'tools'

Change 239645 merged by jenkins-bot:
Remove pywikibot modules if import failed

https://gerrit.wikimedia.org/r/239645

XZise lowered the priority of this task from Unbreak Now! to Low.Sep 20 2015, 2:43 PM

With ba891785 this problem should be largely solved. The issue is simply that when someone imports pywikibot and a RuntimeError in pywikibot.config2 occurs that it successfully imported pywikibot.tools and pywikibot.logging which stay imported. When then something imports pywikibot again it doesn't import and run these two packages so pywikibot itself doesn't get initialized. So by forcing a reimport of those packages this should be fixed.

Change 239640 abandoned by John Vandenberg:
Do not import tools in config

Reason:
ba891785 was the chosen solution; allowing config to use tools seems very sensible.

https://gerrit.wikimedia.org/r/239640

Change 239643 merged by jenkins-bot:
Dont overwrite tools.debug

https://gerrit.wikimedia.org/r/239643

Change 239648 merged by jenkins-bot:
Remove implicit use of pywikibot

https://gerrit.wikimedia.org/r/239648

jayvdb claimed this task.

Solved two different ways, for good measure.

There is the small problem that .pyc files cause problems if one switches from commits after to commits before these fixes.
Not sure what we can, or should, do to resolve that. The passage of time makes this problem go away.