IMO externals approach is not acceptable. That is just recreating hacks which will not be maintained. We need to be using released software.

I've been looking at virtual env's and the conda's , and been experimenting setting them up nicely.

pyenv needs a bit of assistance to work with the conda's (playing with LD_LIBRARY_PATH):

https://github.com/yyuu/pyenv/pull/617

pyenv-virtualenv works very well. See https://github.com/jayvdb/pyenv-virtualenv/commit/b6168315cba0f54e4db2a07f010e8a69501ac0cd
(messy; contains unnecessary code copied from my pyenv experimentation)
results at
https://travis-ci.org/jayvdb/pyenv-virtualenv/builds

Noting there is a current (untriaged) issue with latest pip/setuptools and conda : https://github.com/pypa/setuptools/issues/605

Z360#5459

@jayvdb: So what your conclusion from T135835#2352783 can you give a recommendation on using conda? What about miniconda or anaconda? What's the reason for using pyenv?

[15:22:12] <jayvdb> DrTrigon, the usual way to do that now is to build a custom Docker that build the in-development dependencies
[15:23:05] <DrTrigon> Do you have an example at hand?
[15:24:17] <jayvdb> ya, pywikibot has a good example of something similar -- https://github.com/wikimedia/pywikibot-core/blob/master/setup.py#L105
[15:24:41] <DrTrigon> cool thanks! I'll come back on that later!
[15:24:52] <jayvdb> we use https://github.com/AlereDevices/lunatic-python , as that has some commits that havent been merged into the main library

As far as I can see none of the mentioned packages like vistrails.core.modules.package.Package, libapt and yum or conda is anywhere near the features of pywikibot-compat.externals.

From what I understand Docker is very poppular, has gained a lot of traction recently [1] and should be able to allow us to "package" even the most nasty developping self-compiled dependencies. In very hard cases, e.g. running on another OS than the docker was intended for Vagrant should be able to help out.

[1] http://www.zdnet.com/article/docker-gains-enterprise-devops-traction/

There are 2 "shoulds" in the the statement before, can somebody with experience in these tools comment please?

I have a few questions here, probably due to my lack of experience with externals:

• Could you elaborate on what "on need" means in "can compile code on need" ? I've seen the opencv conda package compile opencv if that's what you mean ?
• What does the column "can install from URL" mean ? How does it know how to install it after downloading the given URL ? i.e. does it use setup.py ? cmake ? etc ? Or does it unzip a file ?

@jayvdb Could you also point us to discussions as to when externals was removed ? I believe they would have made a pro/con list of it, and that would be helpful (for me atleast) to read. I couldn't find any phabricator tasks discussing the decision to remove externals based on a quick search ...

Am 13. Juni 2016 15:38:38 MESZ, schrieb AbdealiJK <no-reply@phabricator.wikimedia.org>:

AbdealiJK added a comment.

I have a few questions here, probably due to my lack of experience with
externals:

Could you elaborate on what "on need" means in "can compile code on
need" ? I've seen the opencv conda package compile opencv if that's

"On need" means basically if a library changed and the c code did crash due to the old missing one. The code tried to recompile (using then the new lib) and te-run. This might be possible to avoid, but it turned out to be pretty useful and robust.

what you mean ?What does the column "can install from URL" mean ? How
does it know how to install it after downloading the given URL ? i.e.
does it use setup.py ? cmake ? etc ? Or does it unzip a file ?

Basically download and unzip a file (e.g. python module) to local directory (per user/installation).

@jayvdb Could you also point us to discussions as to when externals was
removed ? I believe they would have made a pro/con list of it, and that
would be helpful (for me atleast) to read. I couldn't find any
phabricator tasks discussing the decision to remove externals based on
a quick search ...

I doubt there are any. I guess catimages.py was the only script using it, due to me. I think nobody understood or supported 'externals'. So I think they did not care and were quite happy removing it... ;) (I remember some "quite involved" discussions back then, when I was still young and foolish... ;)

Dr. Trigon