Page MenuHomePhabricator

Pinning pywikibot to 3.0.20200111 doesn't work with setuptools
Open, Needs TriagePublic

Description

For those of us stuck on old MediaWiki versions, pywikibot helpfully provides a warning that new versions of pywikibot might not work with our MediaWiki version: https://github.com/wikimedia/pywikibot/blob/3d5f366a465f4e0c3115bffed919b0185ac9feb8/pywikibot/site/__init__.py#L2753-L2760. This indicates that pywikibot version 3.0.20200111 should be used. Unfortunately, pinning the pywikibot version in setup.py via

install_requires=["pywikibot==3.0.20200111"],

is not working reliably because pywikibots setup.py causes the generated version number to be different from 3.0.20200111 when installed via setup.py. Instead, the following warning is generated:

[...]
creating /Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages/pywikibot-3.0.dev0-py3.7.egg
Extracting pywikibot-3.0.dev0-py3.7.egg to /Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages
pywikibot 3.0.dev0 is already the active version in easy-install.pth

Installed /Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages/pywikibot-3.0.dev0-py3.7.egg
error: The 'pywikibot==3.0.20200111' distribution was not found and is required by pwbtest

This doesn't break anything so far. However, if the same setup.py specifies entrypoints for console_scripts, those don't work, either, because they perform a version check before running user code:

Traceback (most recent call last):
  File "/Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (pywikibot 3.0.dev0 (/Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages/pywikibot-3.0.dev0-py3.7.egg), Requirement.parse('pywikibot==3.0.20200111'), {'pwbtest'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/wieland/dev/pwbtest/venv/bin/pwbtestcli", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3250, in <module>
    @_call_aside
  File "/Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3234, in _call_aside
    f(*args, **kwargs)
  File "/Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3263, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pywikibot==3.0.20200111' distribution was not found and is required by pwbtest

I have provided a repository at https://github.com/mineo/pwbtest to reproduce this problem. Run repro.sh in it (it creates a virtualenv in the repository) and follow the instructions.

Installing the same version of pywikibot via pip install pywikibot==3.0.20200111 works, though.

Event Timeline

Mineo created this task.May 27 2020, 6:44 AM
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptMay 27 2020, 6:44 AM
Xqt added a subscriber: Xqt.EditedMay 27 2020, 7:49 AM

mw 1.14-1.18 is still supported with the current release. I think you can Always use "python2" release for older mw versions.

Change 598976 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [doc] Use python2 release for older mw versions

https://gerrit.wikimedia.org/r/598976

Change 409823 had a related patch set uploaded (by Xqt; owner: Xqt):
[pywikibot/core@master] [IMPR] Use one central point for framework version

https://gerrit.wikimedia.org/r/409823

Mineo added a comment.May 27 2020, 9:12 AM

mw 1.14-1.18 is still supported with the current release. I think you can Always use "python2" release for older mw versions.

Is the "python2" release available on PyPI? I found it by navigating from https://doc.wikimedia.org/pywikibot/master/ to https://doc.wikimedia.org/pywikibot/master/installation.html to https://www.mediawiki.org/wiki/Manual:Pywikibot/Installation, which is a bit annoying, compared to just specifying a version number in setup.py.

mw 1.14-1.18 is still supported with the current release. I think you can Always use "python2" release for older mw versions.

I'm connecting to mw 1.16.

Dvorapa added a subscriber: Dvorapa.EditedMay 27 2020, 9:14 AM

The 'pywikibot==3.0.20200111' distribution was not found and is required by pwbtest

This bug was fixed in 3.0.20200508.

Is the "python2" release available on PyPI?

Any 3.0.x release is automatically a "python2" release

This indicates that pywikibot version 3.0.20200111 should be used.

3.0.20200508 still supports old MW versions. And the next 1 or 2 releases will also support old MW versions. Could you test the latest release if any of your described issues still occur?

Mineo added a comment.EditedMay 27 2020, 11:18 AM

Is the "python2" release available on PyPI?

Any 3.0.x release is automatically a "python2" release

Thanks for clarifying & asking for the message change in https://gerrit.wikimedia.org/r/#/c/pywikibot/core/+/598976/ to make this clear.

This indicates that pywikibot version 3.0.20200111 should be used.

3.0.20200508 still supports old MW versions. And the next 1 or 2 releases will also support old MW versions. Could you test the latest release if any of your described issues still occur?

Same thing with 3.0.20200508:

Processing dependencies for pwbtest==0.1.dev2+g9c44c35.d20200527
Searching for pywikibot==3.0.20200508
Reading https://pypi.org/simple/pywikibot/
Downloading https://files.pythonhosted.org/packages/42/3d/ecfce167ec6b29726d043ec324daf29e48b094c37319a7b193bbd9e40610/pywikibot-3.0.20200508.tar.gz#sha256=c61095a8d56c02ebc73f5c23f7e75248f11e0272d88cbe6b5d258b0438cbb0b6
Best match: pywikibot 3.0.20200508
Processing pywikibot-3.0.20200508.tar.gz
Writing /var/folders/pr/6ffcygy56cj18vcwp0njbtyh0000gn/T/easy_install-0wm8hase/pywikibot-3.0.20200508/setup.cfg
Running pywikibot-3.0.20200508/setup.py -q bdist_egg --dist-dir /var/folders/pr/6ffcygy56cj18vcwp0njbtyh0000gn/T/easy_install-0wm8hase/pywikibot-3.0.20200508/egg-dist-tmp-81utx7nq
fatal: Kein Git-Repository (oder irgendeines der Elternverzeichnisse): .git
Command '['git', 'log', '-1', '--format=%ci']' returned non-zero exit status 128.
The 'pywikibot' distribution was not found and is required by the application
Cannot include pywikibot/DIRECTORIES.rst; file not found
Cannot include ROADMAP.rst; file not found
Cannot include CODE_OF_CONDUCT.rst; file not found
zip_safe flag not set; analyzing archive contents...
pywikibot.__pycache__.config2.cpython-37: module references __file__
pywikibot.__pycache__.i18n.cpython-37: module references __file__
pywikibot.__pycache__.i18n.cpython-37: module references __path__
pywikibot.__pycache__.version.cpython-37: module references __file__
creating /Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages/pywikibot-3.0.dev0-py3.7.egg
Extracting pywikibot-3.0.dev0-py3.7.egg to /Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages
Adding pywikibot 3.0.dev0 to easy-install.pth file

Installed /Users/wieland/dev/pwbtest/venv/lib/python3.7/site-packages/pywikibot-3.0.dev0-py3.7.egg
error: The 'pywikibot==3.0.20200508' distribution was not found and is required by pwbtest
Run "source venv/bin/activate" and then pwbtestcli. It will not work

If I'm reading https://github.com/wikimedia/pywikibot/blob/fac57cf2773186669b96b20548876b6a43a7e0fc/setup.py#L171-L187 correctly, the calls to git and get_distribution both fail:

fatal: Kein Git-Repository (oder irgendeines der Elternverzeichnisse): .git
Command '['git', 'log', '-1', '--format=%ci']' returned non-zero exit status 128.
The 'pywikibot' distribution was not found and is required by the application

(The first line is german for "This is not a git repository because '.git' does not exist").

Dvorapa added a comment.EditedMay 27 2020, 11:44 AM

Hmmm, we should use PKGINFO in that case (@Xqt). But we do use it since January. This is weird, why it does not work? What version of setuptools do you use?

The 'pywikibot' distribution was not found and is required by the application

The fallback to PKGINFO seems not working in your case. But why?

Dvorapa added a comment.EditedMay 27 2020, 12:07 PM

Please describe the actual steps and commands you do to make this error occur, I was unable to reproduce the issue so far.

Xqt added a comment.May 27 2020, 2:49 PM

I've no glue what is going wrong, probably a wrong setuptools version?

Maybe we should switch to a fixed versioning like it was proposed by that patch given above.

Mineo added a comment.May 27 2020, 7:25 PM

The 'pywikibot' distribution was not found and is required by the application

The fallback to PKGINFO seems not working in your case. But why?

From a quick look at the setuptools documentation, get_distribution will only work if pywikibot is already installed. That's not the case when installing pywikibot itself.

Please describe the actual steps and commands you do to make this error occur, I was unable to reproduce the issue so far.

See my first post: clone https://github.com/mineo/pwbtest, run repro.sh. Optionally, bump the version number of pywikibot that's required in setup.py.

I've no glue what is going wrong, probably a wrong setuptools version?

I've added pip install --upgrade setuptools to repro.sh before the call to setup.py develop. That installs setuptools 46.4.0, the most recent one. That didn't help.

Dvorapa added a comment.EditedMay 28 2020, 12:36 PM

Okay, so I've tried your repo from https://github.com/mineo/pwbtest . I got the same error, but I noticed you use setup.py develop which is not a recommended practice afaik. So I've tried pip install -e . instead and everything worked like a charm.

Using setup.py develop these days is literally like asking for a trouble from my point of view, it is also not recommended by Python developers. These days PEP 517 and 518 make the use of setup.py install and develop effectively deprecated. Similarly setuptools project makes setup.py test deprecated and test frameworks like pytest, unittest, nosetest etc. also recommend using their way instead. So generally using setup.py as an executable script instead of using it just as a source of package information for package manager (like pip) is not a good idea (except setup.py sdist of course).

Dvorapa added a comment.EditedMay 28 2020, 12:55 PM

I still don't know why exactly pip install -e . works while setup.py develop doesn't. To me those two should work the same, only setup.py develop should first determine package manager in use (we use pip & setuptools) and then call its install function. Anyway, the fix might be easy to do. From your log it seems the distribution for pkg_resources is somewhat incorrect. I would prefer to close this as declined as setup.py develop is known for such bugs like installing the dependencies in an improper way (I found the same issue has been put on hold by googleapis python interface).

Mineo added a comment.May 28 2020, 1:41 PM

Okay, so I've tried your repo from https://github.com/mineo/pwbtest . I got the same error, but I noticed you use setup.py develop which is not a recommended practice afaik. So I've tried pip install -e . instead and everything worked like a charm.

Using setup.py develop these days is literally like asking for a trouble from my point of view, it is also not recommended by Python developers. These days PEP 517 and 518 make the use of setup.py install and develop effectively deprecated.

I can't find any documentation about setup.py develop being deprecated in the setuptools documentation: https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode. The test command in comparison has a big red deprecation banner.

pip install -e and/or using flit install -s as possible workarounds work. Thanks for the pointers.

Mineo added a comment.May 28 2020, 4:17 PM

Hmm, pip install -e . and flit install -s work, python setup.py install does not work. It ends with the same error ("error: The 'pywikibot==3.0.20200508' distribution was not found and is required by pwbtest").

Dvorapa added a comment.EditedMay 28 2020, 5:20 PM

Yeah, that's expected, setup.py install/develop/build will all have the same problem I think. I honestly don't know, how to fix this as I'm currently outside my PC, unable to create the in-the-middle-of setup.py develop conditions to find out, what should setup.py ask if not git or pkg_resources. In dist tarball, the only place with version number is PKGINFO and I don't know, how to instruct setup.py to parse this value from PKGINFO. I'm afraid single-sourced version will either have the same problem, or it would create a new one, but It would definitely be better than the current versioning madness.

I hate this Python packaging and versioning so much, such a good language with such a lame folder structure, packaging and versioning. In comparison to Dart or Go, this is still a madness even with PEP 517 and 518.

Python should totally inspire from them. Simple and clearly defined folder structure, single-sourced all package information and dependencies using a JSON/YAML serialized formats, simple tools to lint, format, build and publish...

Mineo added a comment.May 30 2020, 5:39 PM

I've been using setuptools_scm for years now and have never had any problems with it (I don't maintain any high-profile packages like pywikibot, though). Maybe a switch to it, or something similar, instead of reimplementing versioning in pywikibot itself, could be a solution?

If I got it right, either we would have to get rid of pip install -e option to enable setuptools_scm through pyproject.toml, or we would have to tell people to install setuptools_scm before installing Pywikibot. Both things are quite impractical for larger set-up projects. The issue with pyproject.toml and pip install -e is quite a flaw of PEP 517/518 :/ Although it looks like a good way to solve this.

Change 598976 merged by jenkins-bot:
[pywikibot/core@master] [doc] Use python2 release for older mw versions

https://gerrit.wikimedia.org/r/598976