Page MenuHomePhabricator

Travis python package caching
Closed, DeclinedPublic

Description

Travis allows caching, which we can use to store python package dependencies.
This may help with issues related to downloading packages from pypi and sourceforge, which can sometimes fail. (T106062: Travis builds failing due to 403 downloading pYsearch from sourceforge.net may be an instance of this).

Travis provides a simple mechanism for this, via cache: pip. This depends on the install phase being empty, which was achieved with rPWBC0222b4090dc7: Sync requirements.txt with setup.py.
The result of this can be seen on https://travis-ci.org/jayvdb/pywikibot-core/jobs/70534517#L175

Collecting pYsearch (from -r requirements.txt (line 9))
  pYsearch is potentially insecure and unverifiable.
Collecting google>=1.7 (from -r requirements.txt (line 10))
  Using cached google-1.7-py2-none-any.whl

From that, it seems that only google is cached, but pYsearch isnt. Which may mean pip isnt storing pYsearch in the pip cache, or the travis cacher isnt caching everything in the pip cache.

Details

Related Gerrit Patches:
pywikibot/core : masterEnable pip caching

Event Timeline

jayvdb created this task.Jul 16 2015, 6:18 PM
jayvdb claimed this task.
jayvdb raised the priority of this task from to Needs Triage.
jayvdb updated the task description. (Show Details)
jayvdb added projects: Pywikibot, Pywikibot-tests.
jayvdb added a subscriber: jayvdb.
Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptJul 16 2015, 6:18 PM

iirc, from a week ago, there was a problem with py2.6 caching, and I tried upgrading pip to solve it. https://github.com/jayvdb/pywikibot-core/commit/b5e16ba36ea8c60d3699e00144f01957adc220ba

jayvdb added a comment.EditedJul 17 2015, 3:36 PM

Travis CI # 1531 Make pip use wheels for common packages deals directly with this problem.

Currently, cache: pip isnt an overall benefit for us. After rPWBC0222b4090dc7: Sync requirements.txt with setup.py, our builds are taking approx. 1 hr 45 min. With cache: pip, first build on each branch is 2 hrs, and the second build is approx. 1 hr 50 mins. While caching the downloads does reduce the dependency on the pypi website, it does not eliminate it as pip still downloads the pypi /simple/foo/ page for each package in order to determine the current version to use.

On Python 2.x, using pip version 6.0.7 (default on Travis), only requests and six are cached. Everything else is downloaded, and the install phase takes 60.49s.

Upgrading to pip 7 solves the problem on Python 2.6, causing pyOpenSSL, google, beautifulsoup4, pyparsing, requests_toolbelt & requests_oauthlib to also be reported as cached, and the install phase takes 6.29s. https://travis-ci.org/jayvdb/pywikibot-core/jobs/71417565

Upgrading to pip 7 does not solve the problem on Python 2.7, with the install phase taking 79.29s: https://travis-ci.org/jayvdb/pywikibot-core/jobs/71417558.

I havent worked out why pip on Python 2.7 isnt working properly. We may need to help our dependency projects to work better with pip caching.

One alternative approach is to use pip2pi to do the caching, but IMO these improvements should be made to the travis-ci 'pip' cacher rather than us building our own travis caching infrastructure. However I am not sure whether travis-build and casher is all of the code for the caching infrastructure.

Change 225105 had a related patch set uploaded (by John Vandenberg):
Enable pip caching

https://gerrit.wikimedia.org/r/225105

Also if we do enable pip caching while it is a 'per branch cache', we should probably override the cacher so it only works on the 'wikimedia' account, or only on 'master', as it doesnt have much benefit for developer branches to have their own cache created, as it is rare for additional builds to occur on the same developer branch due to our gerrit workflow.

pip caching not working on Python 2.7. Upstream : https://github.com/travis-ci/travis-ci/issues/4422
pip caching doesnt support dependencies which are in a VCS and not released onto PYPI. Upstream : https://github.com/pypa/pip/issues/218

jayvdb moved this task from Backlog to Reported Upstream on the Upstream board.
AbdealiJK moved this task from Backlog to Upstream on the Pywikibot-tests board.May 5 2016, 3:52 AM

Needs retesting..

We dont use pYsearch any more, and all of our dependencies should now be in pypi.

Also, pip has seen a few releases since this task was created, so maybe the problem is now magically fixed.

Restricted Application added a subscriber: TerraCodes. · View Herald TranscriptMay 5 2016, 8:29 AM
Xqt triaged this task as Low priority.Oct 30 2016, 11:47 AM
Xqt removed jayvdb as the assignee of this task.Apr 14 2018, 4:58 PM
Xqt raised the priority of this task from Low to Medium.
Xqt assigned this task to jayvdb.Apr 14 2018, 5:03 PM

Change 225105 merged by jenkins-bot:
[pywikibot/core@master] Enable pip caching

https://gerrit.wikimedia.org/r/225105

Xqt closed this task as Declined.Apr 14 2018, 7:07 PM
Xqt added a subscriber: Xqt.

Does not shorten test time so that patch could be reverted.