Evaluate devpi for caching Pypi python packages
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Oct 7 2015, 10:42 AM

Description

On pmcache.integration.eqiad.wmflabs investigate usage of devpi as a package manager proxy cache.

Related Objects
Search...

Status	Assigned	Task
Resolved	hashar	T119138 [keyresult] Migrate majority of CI jobs to Nodepool (part 2)
Duplicate	None	T136249 Track what tests we have left to convert to nodepool
Resolved	hashar	T119140 [keyresult] Migrate as many misc CI jobs as possible to Nodepool
Resolved	hashar	T114315 [keyresult] Migrate majority of CI jobs to Nodepool (part 1)
Duplicate	None	T126774 Run 'npm' job with Node 4 (instead of Node 0.10)
Resolved	Krinkle	T129617 Jobs sometimes fail with "/usr/local/bin/npm: No such file or directory"
Resolved	hashar	T119143 Migrate javascript npm CI jobs to Nodepool
Resolved	hashar	T112560 [tracking] Disposable VMs need a cache for package managers
Resolved	hashar	T114871 Evaluate devpi for caching Pypi python packages

Event Timeline

hashar created this task.Oct 7 2015, 10:42 AM

hashar claimed this task.

hashar raised the priority of this task from to Medium.

hashar updated the task description. (Show Details)

hashar added a project: Continuous-Integration-Scaling.

hashar added subscribers: JanZerebecki, zeljkofilipin, dduvall and 2 others.

Based on http://doc.devpi.net/latest/quickstart-pypimirror.html

Setup

role::labs::lvm::srv (to mount instance extended disk space at /srv/)
contint::packages::python (python, pip etc)
pip install devpi-server

$ /usr/local/bin/devpi-server --version
2.3.1

Create user / directory:

adduser --system --home /var/lib/devpi --group devpi
mkdir -p /srv/cache/devpi
chown devpi:devpi /srv/cache/devpi

Startup

$ sudo --set-home --group=devpi --user=devpi  -- \
    devpi-server --start --host=0.0.0.0 --port=3141 --serverdir /srv/cache/devpi
2015-10-07 11:00:25,267 INFO  NOCTX Loading node info from /srv/cache/devpi/.nodeinfo
2015-10-07 11:00:25,267 INFO  NOCTX generated uuid: 838e5a5389ee4bee86ba81b01c4597fa
2015-10-07 11:00:25,268 INFO  NOCTX wrote nodeinfo to: /srv/cache/devpi/.nodeinfo
2015-10-07 11:00:25,284 INFO  NOCTX DB: Creating schema
2015-10-07 11:00:25,299 INFO  [Wtx-1] setting password for user u'root'
2015-10-07 11:00:25,300 INFO  [Wtx-1] created user u'root' with email None
2015-10-07 11:00:25,300 INFO  [Wtx-1] created root user
2015-10-07 11:00:25,300 INFO  [Wtx-1] created root/pypi index
2015-10-07 11:00:25,304 INFO  [Wtx-1] fswriter0: committed: keys: u'.config',u'root/.config'
2015-10-07 11:00:25,311 INFO  Starting new HTTP connection (1): 0.0.0.0
starting background devpi-server at http://0.0.0.0:3141
/srv/cache/devpi/.xproc/devpi-server$ /usr/local/bin/devpi-server --host=0.0.0.0 --port=3141 --serverdir /srv/cache/devpi
process u'devpi-server' started pid=16549
2015-10-07 11:00:25,424 INFO  Starting new HTTP connection (1): 0.0.0.0
...
2015-10-07 11:00:32,663 INFO  Starting new HTTP connection (72): 0.0.0.0
devpi-server process startup detected
logfile is at /srv/cache/devpi/.xproc/devpi-server/xprocess.log

Initial log:

cat /srv/cache/devpi/.xproc/devpi-server/xprocess.log 
2015-10-07 11:00:25,651 INFO  NOCTX Loading node info from /srv/cache/devpi/.nodeinfo
2015-10-07 11:00:25,651 INFO  NOCTX wrote nodeinfo to: /srv/cache/devpi/.nodeinfo
2015-10-07 11:00:25,657 INFO  NOCTX retrieving initial name/serial list
2015-10-07 11:00:25,660 INFO  Starting new HTTPS connection (1): pypi.python.org
2015-10-07 11:00:32,290 INFO  [Wtx0] fswriter1: committed: keys: u'root/pypi/initiallinks'
2015-10-07 11:00:32,552 INFO  NOCTX Found plugin devpi-server-2.3.1 (/usr/local/lib/python2.7/dist-packages).
2015-10-07 11:00:32,552 INFO  NOCTX Found plugin devpi-server-2.3.1 (/usr/local/lib/python2.7/dist-packages).
2015-10-07 11:00:32,597 INFO  NOCTX devpi-server version: 2.3.1
2015-10-07 11:00:32,598 INFO  NOCTX serverdir: /srv/cache/devpi
2015-10-07 11:00:32,598 INFO  NOCTX uuid: 838e5a5389ee4bee86ba81b01c4597fa
2015-10-07 11:00:32,598 INFO  NOCTX serving at url: http://0.0.0.0:3141
2015-10-07 11:00:32,598 INFO  NOCTX bug tracker: https://bitbucket.org/hpk42/devpi/issues
2015-10-07 11:00:32,598 INFO  NOCTX IRC: #devpi on irc.freenode.net
2015-10-07 11:00:32,599 INFO  NOCTX Hit Ctrl-C to quit.
2015-10-07 11:00:32,665 INFO  [req0] GET /

Usage

Used integration-dev as a playground:

virtualenv devpi
cd devpi
. ./bin/activate
pip install -i http://pmcache.integration.eqiad.wmflabs:3141/root/pypi/ PyYAML

On the cache log:

2015-10-07 11:08:32,326 INFO  [req1] GET /root/pypi/PyYAML/
2015-10-07 11:08:32,335 INFO  [req2] GET /root/pypi/+simple/pyyaml
2015-10-07 11:08:32,338 INFO  Starting new HTTPS connection (1): pypi.python.org
2015-10-07 11:08:32,382 INFO  [req2] [Wtx1] fswriter2: committed: keys: 
    u'root/pypi/+f/74c/94a383886519e/PyYAML-3.10.tar.gz',
    u'root/pypi/+f/dfb/573ab41cdfee1/PyYAML-3.10.win32-py2.5.exe',
    u'root/pypi/+f/036/0710722fdc58e/PyYAML-3.10.win32-py3.0.exe',
    u'root/pypi/+f/d52/af84d2cd6ce9b/PyYAML-3.11.win-amd64-py3.1.exe',
    u'root/pypi/+f/538/cd92d63068161/PyYAML-3.11.win-amd64-py2.7.exe',
    u'root/pypi/+f/4c1/a374cb89fa0ff/PyYAML-3.10.win32-py2.6.exe',
    u'root/pypi/+f/b1a/2b30cdf481da4/PyYAML-3.10.zip',
    u'root/pypi/+f/83b/0f47825feac2d/PyYAML-3.11.win32-py2.6.exe',
    u'root/pypi/+f/5c1/4b49d828766c2/PyYAML-3.11.win32-py3.4.exe',
    u'root/pypi/+f/328/bfd749ee42279/PyYAML-3.11.win-amd64-py3.2.exe',
    u'root/pypi/+f/0df/cdae003642dfd/PyYAML-3.11.win-amd64-py3.4.exe',
    u'root/pypi/+f/fc5/82f2f46f6494a/PyYAML-3.10.win32-py3.2.exe',
    u'root/pypi/+f/733/09e5fe73d7d84/PyYAML-3.10.win32-py2.7.exe',
    u'root/pypi/+f/d75/320c7a904731d/PyYAML-3.11.win32-py2.7.exe',
    u'root/pypi/+f/f50/e08ef0fe55178/PyYAML-3.11.tar.gz',
    u'root/pypi/+f/c39/90781cc999744/PyYAML-3.11.win32-py3.3.exe',
    u'root/pypi/+f/89c/bc92cda979042/PyYAML-3.11.zip',
    u'root/pypi/+f/672/275294ababb18/PyYAML-3.11.win-amd64-py3.3.exe',
    u'root/pypi/+f/6c8/30441d9df7034/PyYAML-3.10.win32-py3.1.exe',
    u'root/pypi/+f/1b3/b956bbe638c50/PyYAML-3.11.win32-py3.1.exe',
    u'root/pypi/+f/af7/eade45dcb4084/PyYAML-3.11.win-amd64-py2.6.exe',
    u'root/pypi/+links/pyyaml',
    u'root/pypi/+f/b40/a15302b494d0b/PyYAML-3.11.win32-py3.2.exe'
2015-10-07 11:08:32,436 INFO  [req3] GET /root/pypi/+f/89c/bc92cda979042/PyYAML-3.11.zip
2015-10-07 11:08:32,446 INFO  [req3] [Wtx2] reading remote: https://pypi.python.org/packages/source/P/PyYAML/PyYAML-3.11.zip, target root/pypi/+f/89c/bc92cda979042/PyYAML-3.11.zip
2015-10-07 11:08:32,459 INFO  [req3] [Wtx2] fswriter3: committed: keys: u'root/pypi/+f/89c/bc92cda979042/PyYAML-3.11.zip', files_commit: +files/root/pypi/+f/89c/bc92cda979042/PyYAML-3.11.zip

Back to the client side:

rm -fR devpi
virtualenv devpi
cd devpi
. ./bin/activate
pip install -i http://pmcache.integration.eqiad.wmflabs:3141/root/pypi/ PyYAML

Cache log:

2015-10-07 11:16:19,143 INFO  [req4] GET /root/pypi/PyYAML/
2015-10-07 11:16:19,152 INFO  [req5] GET /root/pypi/+simple/pyyaml
2015-10-07 11:16:19,207 INFO  [req6] GET /root/pypi/+f/89c/bc92cda979042/PyYAML-3.11.zip

Success!

Config template

Passing --gen-config instead of --start generates a set of basic startup scripts:

sudo --set-home --group=devpi --user=devpi  --     devpi-server --gen-config --host=0.0.0.0 --port=3141 --serverdir /srv/cache/devpi
wrote gen-config/supervisor-devpi.conf
wrote gen-config/nginx-devpi.conf
wrote gen-config/crontab
wrote gen-config/net.devpi.plist
wrote gen-config/devpi.service

For systemd that is devpi.service

[Unit]
Requires=network-online.target
After=network-online.target

[Service]
Type=forking
PIDFile=/srv/cache/devpi/.xproc/devpi-server/xprocess.PID
Restart=always
ExecStart=/usr/local/bin/devpi-server --host=0.0.0.0 --port=3141 --serverdir /srv/cache/devpi --start
ExecStop=/usr/local/bin/devpi-server --host=0.0.0.0 --port=3141 --serverdir /srv/cache/devpi --stop
User=devpi

Conclusion

devpi seems to work out of the box. The PyYAML is a native module, so only the tarball is cached and that still requires compilation on each run :-/

The puppet integration should be straightforward.

On the client side we can restore part of https://gerrit.wikimedia.org/r/#/c/226730/ which removed the ~/.pip/pip.conf and inject:

[global]
index-url = http://pmcache.integration.eqiad.wmflabs:3141/root/pypi/+simple/

hashar added a project: Continuous-Integration-Infrastructure.Oct 7 2015, 11:28 AM

hashar set Security to None.

hashar mentioned this in T112560: [tracking] Disposable VMs need a cache for package managers.

hashar mentioned this in T114650: tox-jessie job is failing randomly on integration/config with could not install deps, because of throttling.Oct 7 2015, 12:36 PM

hashar moved this task from Backlog to In-progress on the Continuous-Integration-Scaling board.Oct 7 2015, 8:09 PM

I have completed the evaluation. It works pretty much out of the box.

hashar mentioned this in T117207: Puppetize / play out with devpi.Oct 30 2015, 1:51 PM

Integration being done via T117207

Change 250009 had a related patch set uploaded (by Hashar):
(WIP) use pmcache has a pypi cache

https://gerrit.wikimedia.org/r/250009

gerritbot added a project: Patch-For-Review.Oct 30 2015, 2:42 PM

hashar moved this task from In-progress to Done on the Continuous-Integration-Scaling board.Nov 2 2015, 1:43 PM

greg added a project: OKR-Work.Jan 11 2016, 10:57 PM

Evaluate devpi for caching Pypi python packagesClosed, ResolvedPublicActions