Page MenuHomePhabricator

Meeting about handling non-pip packages / replace externals module
Closed, ResolvedPublic

Description

Date: Week 3 (did it on Week 4)
Time: (we did it on 17th of June)
Type: (was on skype)
Participants: AbdealiJK, DrTrigon, jayvdb

Description: Meeting about handling non-pip packages, see T134121 and T135834

Dependencies to handle: https://commons.wikimedia.org/wiki/User:AbdealiJK/file-metadata/Dependencies

Compare Features:

Package namecan install from os package managementcan install from URLcan apply patchescan compile code on neednotes
pywikibot-compat.externalsyesyesyesyesderived from vistrails.core.modules.package.Package, see below
vistrails.core.modules.package.Packageyesnonono
libapt and yumyesnonono
condano - it IS pack. managm.noyesnot "on need"solve T132659: Conda-based packaging for Windows first in order to then add to a docker image (PAWS or general for install on users machine)
Docker????used by pywikibot
Vagrant????used by https://www.mediawiki.org/wiki/MediaWiki-Vagrant / MediaWiki-Vagrant
  1. can install from os package management: find, download and install packages not handeled by pip - this means handling apt-get, yum and the special case of nothing for win as well as mac
  2. automatic compillation for python modules needing it during install - this also involves re-compilation on need if a new package was updated meanwhile

Agenda:

Minutes of the Meeting:

  • pywikibot-compat.externals, vistrails.core.modules.package.Package and libapt&yum are no options that satisfy our needs
  • Docker is a completely different approach (like virtual machine/onw closed environment) that is used by a lot of other projects e.g. pywikibot already
    • has issues on windows that can be worked around by either using a linux guest in a VM or Vagrant (both common)
    • allows for easy and automated first installation and having a look at the code in action - users can then decide to got down the harder path and install themselves manually (pip)
    • does that by essentially having a install/setup script (similar to travis-ci config files) that does the job
  • conda might come in handy for manual install (pip) on windows where we actually lack a package management at all; conda will allow us to fulfill dependencies anyways

Summary: we go for Docker (with help of e.g. Vagrant and VM like VirtualBox) and conda (mainly for win to fulfill deps)

Event Timeline

DrTrigon created this task.May 20 2016, 1:22 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptMay 20 2016, 1:22 PM
DrTrigon updated the task description. (Show Details)May 20 2016, 1:34 PM
DrTrigon updated the task description. (Show Details)May 23 2016, 1:56 PM
DrTrigon updated the task description. (Show Details)May 23 2016, 2:01 PM
DrTrigon updated the task description. (Show Details)May 24 2016, 10:47 AM
DrTrigon updated the task description. (Show Details)May 27 2016, 1:25 PM

IMO externals approach is not acceptable. That is just recreating hacks which will not be maintained. We need to be using released software.

I've been looking at virtual env's and the conda's , and been experimenting setting them up nicely.

pyenv needs a bit of assistance to work with the conda's (playing with LD_LIBRARY_PATH):

https://github.com/yyuu/pyenv/pull/617

pyenv-virtualenv works very well. See https://github.com/jayvdb/pyenv-virtualenv/commit/b6168315cba0f54e4db2a07f010e8a69501ac0cd
(messy; contains unnecessary code copied from my pyenv experimentation)
results at
https://travis-ci.org/jayvdb/pyenv-virtualenv/builds

jayvdb updated the task description. (Show Details)Jun 3 2016, 11:56 AM
jayvdb updated the task description. (Show Details)
DrTrigon updated the task description. (Show Details)Jun 6 2016, 8:00 AM
DrTrigon updated the task description. (Show Details)Jun 7 2016, 7:37 AM
jayvdb added a comment.EditedJun 10 2016, 5:29 AM

Noting there is a current (untriaged) issue with latest pip/setuptools and conda : https://github.com/pypa/setuptools/issues/605

DrTrigon updated the task description. (Show Details)Jun 10 2016, 1:05 PM
jayvdb updated the task description. (Show Details)Jun 10 2016, 1:11 PM
This comment was removed by DrTrigon.

@jayvdb: So what your conclusion from T135835#2352783 can you give a recommendation on using conda? What about miniconda or anaconda? What's the reason for using pyenv?

DrTrigon updated the task description. (Show Details)Jun 10 2016, 2:44 PM
DrTrigon updated the task description. (Show Details)Jun 10 2016, 2:52 PM
DrTrigon updated the task description. (Show Details)
DrTrigon updated the task description. (Show Details)Jun 10 2016, 2:54 PM

[15:22:12] <jayvdb> DrTrigon, the usual way to do that now is to build a custom Docker that build the in-development dependencies
[15:23:05] <DrTrigon> Do you have an example at hand?
[15:24:17] <jayvdb> ya, pywikibot has a good example of something similar -- https://github.com/wikimedia/pywikibot-core/blob/master/setup.py#L105
[15:24:41] <DrTrigon> cool thanks! I'll come back on that later!
[15:24:52] <jayvdb> we use https://github.com/AlereDevices/lunatic-python , as that has some commits that havent been merged into the main library

DrTrigon updated the task description. (Show Details)Jun 10 2016, 2:57 PM
DrTrigon renamed this task from Meeting about handling non-pip packages to Meeting about handling non-pip packages / replace externals module.Jun 10 2016, 3:01 PM
DrTrigon updated the task description. (Show Details)
DrTrigon updated the task description. (Show Details)Jun 10 2016, 3:05 PM
This comment was removed by DrTrigon.
DrTrigon updated the task description. (Show Details)Jun 12 2016, 8:29 PM
DrTrigon updated the task description. (Show Details)Jun 13 2016, 11:59 AM
DrTrigon updated the task description. (Show Details)Jun 13 2016, 12:32 PM
DrTrigon added a comment.EditedJun 13 2016, 1:01 PM

As far as I can see none of the mentioned packages like vistrails.core.modules.package.Package, libapt and yum or conda is anywhere near the features of pywikibot-compat.externals.

From what I understand Docker is very poppular, has gained a lot of traction recently [1] and should be able to allow us to "package" even the most nasty developping self-compiled dependencies. In very hard cases, e.g. running on another OS than the docker was intended for Vagrant should be able to help out.

[1] http://www.zdnet.com/article/docker-gains-enterprise-devops-traction/

There are 2 "shoulds" in the the statement before, can somebody with experience in these tools comment please?

I have a few questions here, probably due to my lack of experience with externals:

  • Could you elaborate on what "on need" means in "can compile code on need" ? I've seen the opencv conda package compile opencv if that's what you mean ?
  • What does the column "can install from URL" mean ? How does it know how to install it after downloading the given URL ? i.e. does it use setup.py ? cmake ? etc ? Or does it unzip a file ?

@jayvdb Could you also point us to discussions as to when externals was removed ? I believe they would have made a pro/con list of it, and that would be helpful (for me atleast) to read. I couldn't find any phabricator tasks discussing the decision to remove externals based on a quick search ...

DrTrigon added a comment.EditedJun 13 2016, 3:15 PM

Am 13. Juni 2016 15:38:38 MESZ, schrieb AbdealiJK <no-reply@phabricator.wikimedia.org>:

AbdealiJK added a comment.
I have a few questions here, probably due to my lack of experience with
externals:
Could you elaborate on what "on need" means in "can compile code on
need" ? I've seen the opencv conda package compile opencv if that's

"On need" means basically if a library changed and the c code did crash due to the old missing one. The code tried to recompile (using then the new lib) and te-run. This might be possible to avoid, but it turned out to be pretty useful and robust.

what you mean ?What does the column "can install from URL" mean ? How
does it know how to install it after downloading the given URL ? i.e.
does it use setup.py ? cmake ? etc ? Or does it unzip a file ?

Basically download and unzip a file (e.g. python module) to local directory (per user/installation).

@jayvdb Could you also point us to discussions as to when externals was
removed ? I believe they would have made a pro/con list of it, and that
would be helpful (for me atleast) to read. I couldn't find any
phabricator tasks discussing the decision to remove externals based on
a quick search ...

I doubt there are any. I guess catimages.py was the only script using it, due to me. I think nobody understood or supported 'externals'. So I think they did not care and were quite happy removing it... ;) (I remember some "quite involved" discussions back then, when I was still young and foolish... ;)

Dr. Trigon

DrTrigon closed this task as Resolved.Jun 17 2016, 7:01 PM
DrTrigon updated the task description. (Show Details)