Page MenuHomePhabricator

pkg_resources.VersionConflict for pyOpenSSL
Closed, ResolvedPublic

Description

Python 2.7_with_system_site_packages in Trusty build environment of Travis is failing with the following error:

$ if [[ "$USE_NOSE" == "1" ]]; then nosetests --version ; if [[ "$SITE_ONLY" == "1" ]]; then python setup.py nosetests --tests tests --verbosity=2 -a "family=$FAMILY,code=$LANGUAGE" --with-trim --with-coverage --cover-package=. ; else python setup.py nosetests --tests tests --verbosity=2 --with-trim --with-coverage --cover-package=. ; fi ; elif [[ "$USE_PYTEST" == "1" ]]; then if [[ "$SITE_ONLY" == "1" ]]; then python setup.py pytest --addopts="-vvv -s --timeout=$TEST_TIMEOUT --cov=. -a \"family=='$FAMILY' and code=='$LANGUAGE'\"" ; else python setup.py pytest --addopts="-vvv -s --timeout=$TEST_TIMEOUT --cov=." ; fi else coverage run setup.py test ; fi

/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/setuptools/dist.py:285: UserWarning: Normalizing '3.0-dev' to '3.0.dev0'

  normalized_version,

running test

Searching for idna>=2.0.0

Reading https://pypi.python.org/simple/idna/

Best match: idna 2.4

Downloading https://pypi.python.org/packages/a3/06/40cb383eaea6e97047666db51abc2f2b32046f3e2a6e5ab2b946630f6062/idna-2.4.tar.gz#md5=b27328914784bf3e4f6fae16f4b75ba9

Processing idna-2.4.tar.gz

Writing /tmp/easy_install-vjRK2N/idna-2.4/setup.cfg

Running idna-2.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-vjRK2N/idna-2.4/egg-dist-tmp-ehjsp_

warning: no previously-included files matching '*.pyc' found under directory 'tools'

warning: no previously-included files matching '*.pyc' found under directory 'tests'

zip_safe flag not set; analyzing archive contents...

Moving idna-2.4-py2.7.egg to /home/travis/build/wikimedia/pywikibot-core/.eggs

Installed /home/travis/build/wikimedia/pywikibot-core/.eggs/idna-2.4-py2.7.egg

Traceback (most recent call last):

  File "setup.py", line 224, in <module>

    use_2to3=False

  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup

    dist.run_commands()

  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands

    self.run_command(cmd)

  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command

    cmd_obj.run()

  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/setuptools/command/test.py", line 132, in run

    self.distribution.install_requires)

  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/setuptools/dist.py", line 313, in fetch_build_eggs

    replace_conflicting=True,

  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 836, in resolve

    dist = best[req.key] = env.best_match(req, ws, installer)

  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1074, in best_match

    dist = working_set.find(req)

  File "/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 711, in find

    raise VersionConflict(dist, req)

pkg_resources.VersionConflict: (pyOpenSSL 0.13 (/usr/lib/python2.7/dist-packages), Requirement.parse('pyOpenSSL>=0.14'))

https://travis-ci.org/wikimedia/pywikibot-core/jobs/208382609#L731

Details

Related Changes in Gerrit:

Event Timeline

Dalba renamed this task from Travis tests are failing in Trusty build environment to pkg_resources.VersionConflict pyOpenSSL.Mar 7 2017, 9:34 AM
Dalba updated the task description. (Show Details)
Dalba updated the task description. (Show Details)
Dalba renamed this task from pkg_resources.VersionConflict pyOpenSSL to pkg_resources.VersionConflict for pyOpenSSL.Mar 7 2017, 9:42 AM

This particular build uses python: '2.7_with_system_site_packages' in its travis matrix which should be equivalent to virtualenv --system-site-packages.

Trusty is shipped with pyopenssl 0.13.

There is a Precise build that also uses 2.7_with_system_site_packages. Precise pyopenssl version is 0.12. Surprisingly, that build passes successfully with no version conflict error.

I'm suspicious about PYSETUP_TEST_EXTRAS environment variable.

Dalba removed Dalba as the assignee of this task.Mar 8 2017, 9:40 PM

Observations:

PYSETUP_TEST_EXTRAS removes requests[security] from dependencies.

requests[security] requires pyopenssl v 0.14 or later. It used to depend on 0.13, but that has changed nearly four months ago.

Travis is using Python 2.7.6 to run this build.

As a possible a workaround I tried to remove requests[security] when PYSETUP_TEST_EXTRAS is activated, the rational being that it causes version conflict on both trusty and precise. But that did not work.

Then I tried changing if 'PYSETUP_TEST_EXTRAS' in os.environ to if 'PYSETUP_TEST_EXTRAS' in os.environ and 'DIST' not in os.environ, effectively disabling PYSETUP_TEST_EXTRAS inside setup. Same VersionConflict error.

Finally I removed PYSETUP_TEST_EXTRAS from build enviroment. The error was still there.

It seems that PYSETUP_TEST_EXTRAS is not the cause. It's enough to change precise to trusty to get the same error in job#11.

Here is what's happening and how to fix it:

Trusty has pyopenssl 0.13 as system site package.

When using 2.7_with_system_site_packages while PYSETUP_TEST_EXTRAS is enabled, requirements.txt gets installed. But because the required pyopenssl there is set to 0.13, pyopenssl 0.14 will not be installed as it already meets the requirement. This also explains what's happening in the Precise build: the requirement is not met and therefore the latest version of pyopenssl gets installed.

Fix is simple, don't know why I didn't see this sooner, just bump the requirement version.

Change 341969 had a related patch set uploaded (by Dalba):
[pywikibot/core] Update security requirements of requests

https://gerrit.wikimedia.org/r/341969

Change 341969 merged by jenkins-bot:
[pywikibot/core] Update security requirements of requests

https://gerrit.wikimedia.org/r/341969

Thanks to Xqt for reviewing.