I still have to use the awkward work around

import os
os.environ["PYWIKIBOT_NO_USER_CONFIG"]="1"
import pywikibot

which doesn't even help when the app is packaged for mac os. Then this bug is a showstopper.

This is not a bug.

Could you explain a bit more. Probably make a short sample to demonstrate that issue. Installing pywikibot as a site-package works like this:

C:\pwb>python
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pywikibot
>>> pywikibot.__version__
'6.0.1'
>>>

If you then do a

import pywikibot

This will already activate some code which might fail. Even if you do

import os
os.environ["PYWIKIBOT_NO_USER_CONFIG"]="2"

you might end up with an exception. Since code I am publishing might be downloaded by users in lots of different environments this essentially forces each and every user to configure things to avoid the exception which is not feasible. There is a bad need to avoid this problem. Currently i am thinking about something like a

try:
  import pywikbot
except Exception as ex:
  # ignore it or do something about it
  pass

which looks like a very ugly workaround which i would love to avoid and therefore added this issue to hope for an upstream fix.

see https://github.com/WolfgangFahl/py-3rdparty-mediawiki/commit/8ab0bd846d191fa93cfd661c89e2bc769d69f06b for trying this workaround

In T278076#6933794, @Seppl2013 wrote:

If you then do a

import pywikibot

This doesn't help. As shown above importing pywikibot from site-package works without any failing. Please try the same sample as shown and give me the result. What is the full traceback if importing fails. What is your Pywikibot version installed?

Unfortunatelly I am not able to install your package as a site packages because installing pycrypto always fails.

I tried to run the script on a local working and it worked without any failure (ok I've disabled any Crypto importing because it cannot be installed at my Workstation);

C:\pwb\GIT\github\py-mediawiki>python
Python 3.9.0 (tags/v3.9.0:9cf6752, Oct  5 2020, 15:34:40) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from wikibot import wikibot
>>>

@Xqt https://github.com/WolfgangFahl/py-3rdparty-mediawiki/issues/43 has the issue about pycrypto...

Which sample are you talking about? How could i get a traceback if i only try an import without catching the Exception. I am in the process of adding try/catch now and in on case i am using:

if not "Request" in sys.modules:
            from pywikibot.data.api import Request

as a workaround but that all really awkward and outright most ugly. Fortunately the CI doesn't choke yet see e.g. https://github.com/WolfgangFahl/py-3rdparty-mediawiki/actions - and also my Jenkins still seems to run o.k.

In T278076#6934156, @Seppl2013 wrote:

@Xqt https://github.com/WolfgangFahl/py-3rdparty-mediawiki/issues/43 has the issue about pycrypto...

Which sample are you talking about? How could i get a traceback if i only try an import without catching the Exception. I am in the process of adding try/catch now and in on case i am using:

I meant the samle I made in https://phabricator.wikimedia.org/T278076#6933779. Could you please reproduce it in exactly the same way: invoke python as a shell and just import pywikibot. If there is an exception the traceback will be shown then. Currently I've no glue what's going wrong because I cannot reproduce any failure.

The sample

% python
Python 3.8.8 (default, Mar 18 2021, 06:01:57) 
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from wikibot import wikibot
>>>

Doesn't help since this is not the enviroment py2app creates. py2app will call python in a "clean" environment with not configuration files, no environment variables, no nothing. And in that enviroment the example would fail. But there would also be no proper stacktrace. That is the reason why i opened this ticket. My workaround is working but i can't even see the resulting stderr code.

In T278076#6934165, @Seppl2013 wrote:

The sample

% python
Python 3.8.8 (default, Mar 18 2021, 06:01:57) 
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from wikibot import wikibot
>>>

Doesn't help since this is not the enviroment py2app creates. py2app will call python in a "clean" environment with not configuration files, no environment variables, no nothing. And in that enviroment the example would fail. But there would also be no proper stacktrace. That is the reason why i opened this ticket. My workaround is working but i can't even see the resulting stderr code.

Probably this is a py2app issue then. If it uses '/Users/wf/Documents/pyworkspace/scan2wiki/dist/scan2wiki.app/Contents/Resources/lib/python38.zip/pywikibot/families' as a path then pywikibot seems not properly installed. I don't know what py2app does. Can't you use venv instead?

Pretty sure this is a py2app issue. register_families_folder uses os.listdir() which obviously fails for a folder inside a zip file. What to do in this matter?

• find a way to configure py2app in a way to exclude the family folder to be zipped
• find a way to get all files inside family folder which resides inside a zip file other than is.listdir(). Perhaps glob or pathlib works or any other module.
• pywikibot could catch the OsError and skip loading these family files in such case but I am not sure about the consequences
• pywikibot can use a hardcored list of family files residing in family folder to import them directly

What do you think?

I am happy with my workaround for the time being. I also intend to change my app to a webbased app and use py2app to create an app that only communicates with the webbased app. That way i'll avoid any similar problems. Still to fix things upstream the default should be not to throw an exception but simply log the problem by default. So os.listdir might be wrapped in a try catch blog. Only if configured to specifically show start up issues should there be an exception.

Change 674317 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [IMPR] collect family files from zip folder

https://gerrit.wikimedia.org/r/674317

@Seppl2013: Are you able to review my proposal

looks like avoiding the exception
I do not know how to use the review tool. I logged in but have no clue what to do. I tried edit/marked as reviewed.

In T278076#6937953, @Seppl2013 wrote:

looks like avoiding the exception

Can you test your script with the revised config2.py?

I do not know how to use the review tool. I logged in but have no clue what to do. I tried edit/marked as reviewed.

There should be a CODE-REVIEW+1 button on topleft side of the gerrit page. Or press the REPLY button to make some comments.

Thx. for the explanation. And no - testing the script just now with just the config2.py changed will be tough since the three CI pipelines i am using are all pulling directly from PyPi. For my local environment it would be sort of feasible but i fear the installation routines would override my local changes as they did when i was trying things out for almost a day recently. My current project priorities are also different -i need to make sure i don't have showstoppers and this one isn't any more.

Ok I see. I've tested these changes already on my local repository with pywikibot installed as a site-package and the family file stored in a zip archive subfolder. I think it should work. I expect to deploy the next stable release to pypi in about two weeks. Do you need it earlier?

@Xqt - thx for the reply and asking - i am fine with this schedule - thank you very much.

Change 674317 merged by jenkins-bot:
[pywikibot/core@master] [IMPR] collect family files from zip folder

https://gerrit.wikimedia.org/r/674317