Page MenuHomePhabricator
Create Task
Maniphest T89142

Create a Debian package for NodePool on Debian Jessie
Closed, ResolvedPublic
Actions

Description

The Continuous-Integration-Scaling project, we need OpenStack NodePool to be shipped as a Debian package. Using pip simply does not work and is prone to errors.

The Debian ITP is http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=781027

Obsolete list of dependencies, we are aiming at Debian/Jessie

Requirements from master @ ddd6003 and whether we have them in WMF Debian Jessie apt repo (as of Apr 15th 2015):~~

Python moduleJessie
pbr>=0.6,!=0.7,<1.0 0.8.2
gear 0.5.5
PyYAML 3.11
python-jenkins 0.2.1
paramiko>1.9.0 1.15.1
python-daemon>=2.0.4 1.5.5 , should be fine
extras 0.0.3
statsd>=1.0.0,<3.0 2.0.1
apscheduler>=2.1.1,<3.0 2.1.2
sqlalchemy>=0.8.2,<0.9.0 Not met 0.9.8 , upstream to bump upper bound see below
pyzmq>=13.1.0,<14.0.0 Not met 14.4.0 , upstream to bump upper bound see below
python-glanceclient 1:0.12.0
python-keystoneclient 1:0.10.1
python-novaclient 1:2.18.0
PyMySQL need backport from unstable to jessie-wikimedia T96131
PrettyTable>=0.6,<0.8 0.7.2
six>=1.7.0 1.8.0
diskimage-builder 0.1.30
voluptuous 0.8.2

Some of the requirements will most probably conflicts with the packages available for Trusty on apt.wikimedia.org :-/

Details

SubjectRepoBranchLines +/-
Initial Debian packagingoperations/debs/nodepooldebian+155 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Feb 10 2015, 3:40 PM
hashar raised the priority of this task from to Needs Triage.
hashar updated the task description. (Show Details)
hashar added projects: Upstream, Continuous-Integration-Scaling, Continuous-Integration-Infrastructure, acl*sre-team.
hashar subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 10 2015, 3:40 PM
hashar updated the task description. (Show Details)Feb 10 2015, 4:03 PM
hashar set Security to None.
hashar added a comment.Feb 11 2015, 9:22 PM

I have poked the internal OPS list about creating Debian packages for python software that have conflicting or missing dependencies. Pasted at P284 follow up on OPS list.

hashar mentioned this in P284 [Ops] python packaging and dependencies hell.Feb 11 2015, 9:25 PM
Liuxinyu970226 subscribed.Feb 12 2015, 11:18 AM
Dzahn triaged this task as Medium priority.Feb 13 2015, 9:36 PM
Dzahn subscribed.
Krinkle moved this task from Untriaged to Next on the Continuous-Integration-Infrastructure board.Mar 2 2015, 2:50 PM
hashar updated the task description. (Show Details)Mar 23 2015, 2:28 PM

I have filled a Debian intent to package: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=781027

hashar updated the task description. (Show Details)Mar 24 2015, 8:54 AM
Krenair moved this task from Backlog to Reported Upstream on the Upstream board.Mar 29 2015, 8:22 PM
gerritbot subscribed.Apr 13 2015, 9:26 PM

Change 203961 had a related patch set uploaded (by Hashar):
Initial Debian packaging

https://gerrit.wikimedia.org/r/203961

gerritbot added a project: Patch-For-Review.Apr 13 2015, 9:26 PM
hashar moved this task from Backlog to In-progress on the Continuous-Integration-Scaling board.Apr 13 2015, 9:27 PM
hashar added a parent task: T89143: Puppetize Nodepool configuration.
hashar mentioned this in T89143: Puppetize Nodepool configuration.
hashar added subscribers: Andrew, • chasemp.Apr 13 2015, 10:22 PM

I have created the Gerrit repository operations/debs/nodepool. Pushed some upstream code in the upstream branch and proposed a first debianization https://gerrit.wikimedia.org/r/203961

Result looks promising against Debian Jessie:

P513 dpkg --info nodepool as of https://gerrit.wikimedia.org/r/#/c/203961/3
1# dpkg --info ../nodepool_0.0.1-104-gddd6003_amd64.deb
2 new debian package, version 2.0.
3 size 44786 bytes: control archive=1966 bytes.
4 905 bytes, 15 lines control
5 2881 bytes, 32 lines md5sums
6 156 bytes, 9 lines * postinst #!/bin/sh
7 251 bytes, 14 lines * prerm #!/bin/sh
8 Package: nodepool
9 Version: 0.0.1-104-gddd6003
10 Architecture: amd64
11 Maintainer: Antoine Musso <hashar@free.fr>
12 Installed-Size: 301
13 Pre-Depends: python2.7
14 Depends: python, python-apscheduler, python-diskimage-builder, python-extras, python-gear, python-glanceclient, python-jenkins, python-keystoneclient, python-novaclient, python-paramiko, python-pbr, python-prettytable, python-six, python-sqlalchemy, python-statsd, python-voluptuous, python-yaml, python-zmq, python:any (<< 2.8), python:any (>= 2.7.5-5~)
15 Section: python
16 Priority: optional
17 Homepage: http://ci.openstack.org/nodepool/
18 Description: Launch single use test nodes on OpenStack
19 Nodepool is a system for launching single-use test nodes on demand based on
20 images built with cached data. It is designed to work with any OpenStack based
21 cloud, and is part of a suite of tools that form a comprehensive test system
22 including Jenkins and Zuul.

P514 dpkg -c nodepool as of https://gerrit.wikimedia.org/r/#/c/203961/3
1# dpkg -c ../nodepool_0.0.1-104-gddd6003_amd64.deb
2drwxr-xr-x root/root 0 2015-04-14 00:05 ./
3drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/
4drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/lib/
5drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/lib/python2.7/
6drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/
7drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool-0.0.1.egg-info/
8-rw-r--r-- root/root 3860 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool-0.0.1.egg-info/PKG-INFO
9-rw-r--r-- root/root 1 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool-0.0.1.egg-info/dependency_links.txt
10-rw-r--r-- root/root 1 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool-0.0.1.egg-info/not-zip-safe
11-rw-r--r-- root/root 100 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool-0.0.1.egg-info/entry_points.txt
12-rw-r--r-- root/root 9 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool-0.0.1.egg-info/top_level.txt
13-rw-r--r-- root/root 310 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool-0.0.1.egg-info/requires.txt
14drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool/
15-rw-r--r-- root/root 708 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/version.py
16-rw-r--r-- root/root 13786 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/nodedb.py
17drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool/tests/
18-rw-r--r-- root/root 17104 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/tests/test_allocator.py
19-rw-r--r-- root/root 8317 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/tests/__init__.py
20-rw-r--r-- root/root 11288 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/tests/test_nodepool.py
21-rw-r--r-- root/root 5186 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/tests/test_commands.py
22-rw-r--r-- root/root 1674 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/tests/test_config_validator.py
23-rw-r--r-- root/root 98968 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/nodepool.py
24drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/lib/python2.7/dist-packages/nodepool/cmd/
25-rw-r--r-- root/root 5103 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/cmd/nodepoold.py
26-rw-r--r-- root/root 3620 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/cmd/config_validator.py
27-rw-r--r-- root/root 0 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/cmd/__init__.py
28-rw-r--r-- root/root 15182 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/cmd/nodepoolcmd.py
29-rw-r--r-- root/root 0 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/__init__.py
30-rw-r--r-- root/root 15964 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/allocation.py
31-rw-r--r-- root/root 3271 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/task_manager.py
32-rw-r--r-- root/root 6317 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/fakeprovider.py
33-rw-r--r-- root/root 4541 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/jenkins_manager.py
34-rw-r--r-- root/root 2095 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/nodeutils.py
35-rw-r--r-- root/root 21403 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/provider_manager.py
36-rw-r--r-- root/root 2088 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/sshclient.py
37-rw-r--r-- root/root 4599 2015-04-07 21:55 ./usr/lib/python2.7/dist-packages/nodepool/myjenkins.py
38drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/share/
39drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/share/doc/
40drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/share/doc/nodepool/
41-rw-r--r-- root/root 166 2015-04-13 23:40 ./usr/share/doc/nodepool/changelog.Debian.gz
42-rw-r--r-- root/root 1161 2015-04-13 23:40 ./usr/share/doc/nodepool/copyright
43-rw-r--r-- root/root 2434 2015-04-07 21:55 ./usr/share/doc/nodepool/README.rst
44drwxr-xr-x root/root 0 2015-04-14 00:05 ./usr/bin/
45-rwxr-xr-x root/root 163 2015-04-14 00:05 ./usr/bin/nodepool
46-rwxr-xr-x root/root 161 2015-04-14 00:05 ./usr/bin/nodepoold

Follow up on the Gerrit change.

hashar renamed this task from Create a Debian package for NodePool to Create a Debian package for NodePool on Debian Jessie.Apr 13 2015, 10:24 PM
hashar updated the task description. (Show Details)
hashar updated the task description. (Show Details)
gerritbot added a comment.Apr 14 2015, 8:23 PM

Change 203961 had a related patch set uploaded (by Hashar):
Initial Debian packaging

https://gerrit.wikimedia.org/r/203961

hashar added a comment.Apr 14 2015, 9:13 PM

Some upstream requirements are not matched by Jessie:

UpstreamJessie
python-daemon>=2.0.41.5.5
pyzmq>=13.1.0,<14.0.014.4.0
sqlalchemy>=0.8.2,<0.9.00.9.8

python-daemon had broken 2.0.x released. https://review.openstack.org/#/c/150256/ changed it from <2.0 to >=2.0.4. Potentially nodepool would work just fine with <2.0.

I have poked upstream about it http://lists.openstack.org/pipermail/openstack-infra/2015-April/002639.html

hashar added a comment.Apr 15 2015, 9:26 AM

From upstream at http://lists.openstack.org/pipermail/openstack-infra/2015-April/002646.html

python-daemon 1.5.5 should work
Tests pass with upper bound limits removed and I proposed a patch to soften the limit https://review.openstack.org/#/c/173731/

hashar updated the task description. (Show Details)Apr 15 2015, 10:25 AM
hashar updated the task description. (Show Details)

I have updated the dependency table in the task details to take in account Jessie instead of Trusty.

hashar added a comment.Apr 16 2015, 8:27 PM

I have build the package based on https://gerrit.wikimedia.org/r/#/c/203961/ patchset 7 and uploaded the resulting files at http://people.wikimedia.org/~hashar/debs/nodepool/ which can be grabbed via: terbium.eqiad.wmnet:/home/hashar/public_html/debs/nodepool/

I have installed it on labnodepool1001.eqiad.wmnet to give it a try.

hashar mentioned this in T95045: install/deploy labnodepool1001.Apr 16 2015, 8:29 PM
hashar added a comment.Apr 17 2015, 7:31 PM

During our Apr 17th checkin, we pointed out a new .deb package targeting Jessie should use systemd instead of init script. @hashar is to poke European ops about it.

Andrew closed subtask T96131: Provide Debian package python-pymysql for jessie-wikimedia as Resolved.Apr 17 2015, 9:51 PM
hashar added a project: Nodepool.Apr 21 2015, 9:42 AM
hashar mentioned this in rODEV66b0eb4bb713: Initial Debian packaging.Apr 21 2015, 9:59 AM
hashar added a comment.Apr 22 2015, 3:09 PM

A status update:

The initial Debian packaging is ready for review https://gerrit.wikimedia.org/r/#/c/203961/ patchset 8

I have created a custom patch https://gerrit.wikimedia.org/r/#/c/205564/ which I proposed upstream. It is integrated as a quilt patch with https://gerrit.wikimedia.org/r/#/c/205571/ . That last change let us produce a nodepool_0.0.1-104-gddd6003-wmf2_amd64.deb package which I have deployed on labnodepool1001.eqiad.wmnet.

I have uploaded the files on terbium:

hashar mentioned this in T96867: Use systemd for Nodepool.Apr 22 2015, 3:12 PM
gerritbot added a comment.Apr 24 2015, 7:46 PM

Change 203961 merged by Andrew Bogott:
Initial Debian packaging

https://gerrit.wikimedia.org/r/203961

hashar added a comment.Apr 24 2015, 7:58 PM

We now have a preliminary Debian package which is good enough. We will improve it as we get more familiar with Nodepool.

Switching to systemd is T96867.

hashar lowered the priority of this task from Medium to Low.Apr 24 2015, 7:59 PM
hashar removed projects: Patch-For-Review, Upstream.
hashar moved this task from In-progress to Done on the Continuous-Integration-Scaling board.
hashar moved this task from Done to In-progress on the Continuous-Integration-Scaling board.
hashar mentioned this in rODEVeba07110a1e8: Initial Debian packaging.Apr 24 2015, 8:33 PM
Liuxinyu970226 unsubscribed.Apr 25 2015, 12:17 AM
hashar closed this task as Resolved.Apr 28 2015, 8:46 PM
hashar claimed this task.

The preliminary work is completed. The package will be enhanced with time.

hashar moved this task from In-progress to Done on the Continuous-Integration-Scaling board.Apr 28 2015, 8:46 PM
hashar added a comment.Apr 28 2015, 8:53 PM

I have poked the Debian ITP pointing to our Git/Gerrit repos.

hashar moved this task from Next to Done on the Continuous-Integration-Infrastructure board.Apr 28 2015, 8:54 PM
hashar closed subtask T96867: Use systemd for Nodepool as Resolved.Jul 22 2015, 2:59 PM
Restricted Application added a subscriber: Matanya. · View Herald TranscriptJul 22 2015, 2:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL