Page MenuHomePhabricator

Create a Debian package for NodePool on Debian Jessie
Closed, ResolvedPublic

Description

The Continuous-Integration-Scaling project, we need OpenStack NodePool to be shipped as a Debian package. Using pip simply does not work and is prone to errors.

The Debian ITP is http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=781027

Obsolete list of dependencies, we are aiming at Debian/Jessie

Requirements from master @ ddd6003 and whether we have them in WMF Debian Jessie apt repo (as of Apr 15th 2015):~~

Python moduleJessie
pbr>=0.6,!=0.7,<1.0 0.8.2
gear 0.5.5
PyYAML 3.11
python-jenkins 0.2.1
paramiko>1.9.0 1.15.1
python-daemon>=2.0.4 1.5.5 , should be fine
extras 0.0.3
statsd>=1.0.0,<3.0 2.0.1
apscheduler>=2.1.1,<3.0 2.1.2
sqlalchemy>=0.8.2,<0.9.0 Not met 0.9.8 , upstream to bump upper bound see below
pyzmq>=13.1.0,<14.0.0 Not met 14.4.0 , upstream to bump upper bound see below
python-glanceclient 1:0.12.0
python-keystoneclient 1:0.10.1
python-novaclient 1:2.18.0
PyMySQL need backport from unstable to jessie-wikimedia T96131
PrettyTable>=0.6,<0.8 0.7.2
six>=1.7.0 1.8.0
diskimage-builder 0.1.30
voluptuous 0.8.2

Some of the requirements will most probably conflicts with the packages available for Trusty on apt.wikimedia.org :-/

Details

Related Gerrit Patches:
operations/debs/nodepool : debianInitial Debian packaging

Event Timeline

hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added a subscriber: hashar.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 10 2015, 3:40 PM
hashar updated the task description. (Show Details)Feb 10 2015, 4:03 PM
hashar set Security to None.

I have poked the internal OPS list about creating Debian packages for python software that have conflicting or missing dependencies. Pasted at P284 follow up on OPS list.

Dzahn triaged this task as Medium priority.Feb 13 2015, 9:36 PM
Dzahn added a subscriber: Dzahn.
hashar updated the task description. (Show Details)Mar 24 2015, 8:54 AM
Krenair moved this task from Backlog to Reported Upstream on the Upstream board.Mar 29 2015, 8:22 PM

Change 203961 had a related patch set uploaded (by Hashar):
Initial Debian packaging

https://gerrit.wikimedia.org/r/203961

I have created the Gerrit repository operations/debs/nodepool. Pushed some upstream code in the upstream branch and proposed a first debianization https://gerrit.wikimedia.org/r/203961

Result looks promising against Debian Jessie:

1# dpkg --info ../nodepool_0.0.1-104-gddd6003_amd64.deb
2 new debian package, version 2.0.
3 size 44786 bytes: control archive=1966 bytes.
4 905 bytes, 15 lines control
5 2881 bytes, 32 lines md5sums
6 156 bytes, 9 lines * postinst #!/bin/sh
7 251 bytes, 14 lines * prerm #!/bin/sh
8 Package: nodepool
9 Version: 0.0.1-104-gddd6003
10 Architecture: amd64
11 Maintainer: Antoine Musso <hashar@free.fr>
12 Installed-Size: 301
13 Pre-Depends: python2.7
14 Depends: python, python-apscheduler, python-diskimage-builder, python-extras, python-gear, python-glanceclient, python-jenkins, python-keystoneclient, python-novaclient, python-paramiko, python-pbr, python-prettytable, python-six, python-sqlalchemy, python-statsd, python-voluptuous, python-yaml, python-zmq, python:any (<< 2.8), python:any (>= 2.7.5-5~)
15 Section: python
16 Priority: optional
17 Homepage: http://ci.openstack.org/nodepool/
18 Description: Launch single use test nodes on OpenStack
19 Nodepool is a system for launching single-use test nodes on demand based on
20 images built with cached data. It is designed to work with any OpenStack based
21 cloud, and is part of a suite of tools that form a comprehensive test system
22 including Jenkins and Zuul.

1# dpkg -c ../nodepool_0.0.1-104-gddd6003_amd64.deb
2drwxr-xr-x root/root 0 2015-04-14 00:05 ./
3drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/
4drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/lib/
5drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/lib/python2.7/
6drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/
7drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool-0.0.1.egg-info/
8-rw-r--r-- root/root 3860 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool-0.0.1.egg-info/PKG-INFO
9-rw-r--r-- root/root 1 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool-0.0.1.egg-info/dependency_links.txt
10-rw-r--r-- root/root 1 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool-0.0.1.egg-info/not-zip-safe
11-rw-r--r-- root/root 100 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool-0.0.1.egg-info/entry_points.txt
12-rw-r--r-- root/root 9 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool-0.0.1.egg-info/top_level.txt
13-rw-r--r-- root/root 310 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool-0.0.1.egg-info/requires.txt
14drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool/
15-rw-r--r-- root/root 708 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/version.py
16-rw-r--r-- root/root 13786 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/nodedb.py
17drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool/tests/
18-rw-r--r-- root/root 17104 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/tests/test_allocator.py
19-rw-r--r-- root/root 8317 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/tests/__init__.py
20-rw-r--r-- root/root 11288 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/tests/test_nodepool.py
21-rw-r--r-- root/root 5186 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/tests/test_commands.py
22-rw-r--r-- root/root 1674 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/tests/test_config_validator.py
23-rw-r--r-- root/root 98968 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/nodepool.py
24drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool/cmd/
25-rw-r--r-- root/root 5103 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/cmd/nodepoold.py
26-rw-r--r-- root/root 3620 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/cmd/config_validator.py
27-rw-r--r-- root/root 0 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/cmd/__init__.py
28-rw-r--r-- root/root 15182 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/cmd/nodepoolcmd.py
29-rw-r--r-- root/root 0 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/__init__.py
30-rw-r--r-- root/root 15964 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/allocation.py
31-rw-r--r-- root/root 3271 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/task_manager.py
32-rw-r--r-- root/root 6317 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/fakeprovider.py
33-rw-r--r-- root/root 4541 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/jenkins_manager.py
34-rw-r--r-- root/root 2095 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/nodeutils.py
35-rw-r--r-- root/root 21403 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/provider_manager.py
36-rw-r--r-- root/root 2088 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/sshclient.py
37-rw-r--r-- root/root 4599 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/myjenkins.py
38drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/share/
39drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/share/doc/
40drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/share/doc/nodepool/
41-rw-r--r-- root/root 166 2015-04-13 23:40 ./usr/share/doc/nodepool/changelog.Debian.gz
42-rw-r--r-- root/root 1161 2015-04-13 23:40 ./usr/share/doc/nodepool/copyright
43-rw-r--r-- root/root 2434 2015-04-07 21:55 ./usr/share/doc/nodepool/README.rst
44drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/bin/
45-rwxr-xr-x root/root 163 2015-04-14 00:05 ./usr/bin/nodepool
46-rwxr-xr-x root/root 161 2015-04-14 00:05 ./usr/bin/nodepoold

Follow up on the Gerrit change.

hashar renamed this task from Create a Debian package for NodePool to Create a Debian package for NodePool on Debian Jessie.Apr 13 2015, 10:24 PM
hashar updated the task description. (Show Details)
hashar updated the task description. (Show Details)

Change 203961 had a related patch set uploaded (by Hashar):
Initial Debian packaging

https://gerrit.wikimedia.org/r/203961

Some upstream requirements are not matched by Jessie:

UpstreamJessie
python-daemon>=2.0.41.5.5
pyzmq>=13.1.0,<14.0.014.4.0
sqlalchemy>=0.8.2,<0.9.00.9.8

python-daemon had broken 2.0.x released. https://review.openstack.org/#/c/150256/ changed it from <2.0 to >=2.0.4. Potentially nodepool would work just fine with <2.0.

I have poked upstream about it http://lists.openstack.org/pipermail/openstack-infra/2015-April/002639.html

From upstream at http://lists.openstack.org/pipermail/openstack-infra/2015-April/002646.html

python-daemon 1.5.5 should work
Tests pass with upper bound limits removed and I proposed a patch to soften the limit https://review.openstack.org/#/c/173731/

hashar updated the task description. (Show Details)Apr 15 2015, 10:25 AM
hashar updated the task description. (Show Details)

I have updated the dependency table in the task details to take in account Jessie instead of Trusty.

I have build the package based on https://gerrit.wikimedia.org/r/#/c/203961/ patchset 7 and uploaded the resulting files at http://people.wikimedia.org/~hashar/debs/nodepool/ which can be grabbed via: terbium.eqiad.wmnet:/home/hashar/public_html/debs/nodepool/

I have installed it on labnodepool1001.eqiad.wmnet to give it a try.

During our Apr 17th checkin, we pointed out a new .deb package targeting Jessie should use systemd instead of init script. @hashar is to poke European ops about it.

A status update:

The initial Debian packaging is ready for review https://gerrit.wikimedia.org/r/#/c/203961/ patchset 8

I have created a custom patch https://gerrit.wikimedia.org/r/#/c/205564/ which I proposed upstream. It is integrated as a quilt patch with https://gerrit.wikimedia.org/r/#/c/205571/ . That last change let us produce a nodepool_0.0.1-104-gddd6003-wmf2_amd64.deb package which I have deployed on labnodepool1001.eqiad.wmnet.

I have uploaded the files on terbium:

Change 203961 merged by Andrew Bogott:
Initial Debian packaging

https://gerrit.wikimedia.org/r/203961

We now have a preliminary Debian package which is good enough. We will improve it as we get more familiar with Nodepool.

Switching to systemd is T96867.

hashar lowered the priority of this task from Medium to Low.Apr 24 2015, 7:59 PM
hashar moved this task from In-progress to Done on the Continuous-Integration-Scaling board.
hashar moved this task from Done to In-progress on the Continuous-Integration-Scaling board.
hashar closed this task as Resolved.Apr 28 2015, 8:46 PM
hashar claimed this task.

The preliminary work is completed. The package will be enhanced with time.

I have poked the Debian ITP pointing to our Git/Gerrit repos.

Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 22 2015, 2:59 PM