Page MenuHomePhabricator

Creating an ItemPage object can cause a deadlock
Closed, DeclinedPublic

Description

The line https://github.com/mineo/mb2wikidatabot/blob/master/bot/const.py#L36 sometimes causes the whole python interpreter to block, so it doesn't even exit on ^C anymore. Since this might be a very specific problem related to the order things are imported in etc., here's a rough overview of how that line is reached.

https://github.com/mineo/mb2wikidatabot/blob/master/run.py is the main entry point of the bot, which imports
https://github.com/mineo/mb2wikidatabot/blob/master/bot/common.py, which (in line 8) imports the const.py file.

I've added print statements around every statement in const.py and the one before the creation of the ItemPage is the last one that gets executed.

I have now found a reliable way to fix this and that is the execution of a file that contains only the following code (yes, this is basically a variation of the const.py file):

import pywikibot as wp
WIKIDATA = wp.Site('wikidata', 'wikidata')
WIKIDATA_DATASITE = WIKIDATA.data_repository()
MUSICBRAINZ_WIKIDATAPAGE = wp.ItemPage(WIKIDATA_DATASITE, "Q14005")
MUSICBRAINZ_CLAIM = wp.Claim(WIKIDATA_DATASITE, "P248")

Other ways to fix it include deletions of most files in the pywikibot user directory and/or random invokations of pywikibots scripts/login.py script.

This happens on Python 2.6.5 on Ubuntu 10.04.4 in a virtual environment. This has happened over the last few year every once in a while with different revisions of pywikibot, but I feel like it's now happening way more often. The currently used pywikibot version is 6b2e416b1bd67d16cf70f3c02420bd9124217a13.

Event Timeline

Mineo raised the priority of this task from to High.
Mineo updated the task description. (Show Details)
Mineo added a project: Pywikibot.
Mineo subscribed.
Restricted Application added subscribers: Aklapper, Unknown Object (MLST). · View Herald TranscriptJan 28 2015, 11:08 AM

This might be the same problem as T71781 ; bot initialisation needs to be done after import is completed. @valhallasw had a very nice graph of the import sequence which shows how the deadlock occurs. We should emit a warning somewhere before the deadlock when we know the deadlock will occur.

Is there some place I can read more about the cause of this? At the moment it's not clear to me what you mean by "bot initialisation needs to be done after import is completed" - at which point is the import considered completed (or not)?

jayvdb lowered the priority of this task from High to Low.Jun 6 2015, 10:29 AM

The underlying problem was fixed in master indirectly by T98671.

It still occurs on the 2.0 branch. The solution is to not put code that calls pywikibot outside of a function. i.e. create a main() and do all initialisation code in that.

zhuyifei1999 subscribed.

Pywikibot 2.0 no longer supported T169734