Page MenuHomePhabricator

[spike] investigate the using python wheels for dependencies in prod
Closed, ResolvedPublic

Description

Generally, operations does not want us to use pip & pypi in production. One alterative is to distribute dependencies as a set of python eggs. However, installing a package via an egg results in arbitrary code execution. Recently, we have been considering using Debian files to capture our dependencies, but that proved to be overly complex. Python wheels (binary distribution) provide an alternative to eggs that does not require arbitrary code execution. So, it would be great if we could use wheels instead.

This task is done when someone tries to build all of the [dependencies for running ores-wikimedia-config](https://github.com/wiki-ai/ores-wikimedia-config/blob/master/requirements.txt) and reports how it works out.

Python wheels: http://pythonwheels.com/

Note that building wheels is trivial for pure-python libraries. How hard is it to generate wheels for sklearn, scipy, numpy, etc?

Event Timeline

Halfak renamed this task from [spike] investigate the process for creating a wheels for clib-based dependencies to [spike] investigate the using python wheels for dependencies in prod.Mar 1 2016, 8:08 PM
Halfak updated the task description. (Show Details)
Halfak updated the task description. (Show Details)

I built all the wheels, and can get ores dev_server to start with it (haven't actually tried using it, but I suppose it starting is good enough?).

Things to note:

  • needs a newer version of pip than available on debian
  • pip3 wheels --wheel-dir=<path> --process-dependency-links -r requirements.txt will generate all the required wheels for requirements.txt into the <path>
  • --process-dependency-links is deprecated - so we might need to make the requirements.txt in ores-wikimedia-config fully recursive (the output of pip freeze)
  • we should also consider adding hashes to packages in requirements.txt for extra security and reproducibiltiy.

Have confirmed that we need --process-dependency-links is both needed and deprecated. We should make the ores-wikimedia-config requirements.txt fully recursive.

I

T129109 is outcome of this spike.