Page MenuHomePhabricator

Create a Debian package for Zuul
Closed, ResolvedPublic

Description

Zuul is deployed form a git repository integration/zuul.git We should make it a proper Debian package.

Upstream bug:

https://bugs.launchpad.net/zuul/+bug/1160076

Requirements from master and whether we have them in WMF Ubuntu Trusty apt repo (as of Feb 10th 2015):

pbr>=0.5.21,<1.0 0.7.0
argparse built in python 2.7
PyYAML>=3.1.0 3.10
Paste 1.7.5.1
WebOb>=1.2.3,<1.3 1.3.1
paramiko>=1.8.0 1.10.1
GitPython>=0.3.2.1 0.3.2~RC1
ordereddictbuilt in python 2.7
python-daemon>=2.0.4 Not met 1.5.5
extras 0.0.3
statsd>=1.0.0,<3.0 Not met 3.0.1
voluptuous>=0.7 Not met 0.8.2
gear>=0.5.4,<1.0.0 Not met 0.4.0 (met in Precise)
apscheduler>=2.1.1,<3.0 2.1.2
PrettyTable>=0.6,<0.8 0.7.2
babel>=1.0 1.3
six>=1.6.0 1.9.0 (got backported recently)

Note we have Zuul installed already on Precise but with a lagged out version.

See Also:
https://launchpad.net/bugs/1160076

Details

Reference
bz46552
Related Gerrit Patches:
integration/zuul : patch-queue/debian/precise-wikimediaPackage python deps with dh-virtualenv
operations/puppet : productionzuul: switch install to a Debian package
integration/zuul : debian/trusty-wikimediaForward port precise dh-virtualenv to trusty
integration/zuul : debian/trusty-wikimediaPackage python deps with dh-virtualenv
integration/zuul : debian/precise-wikimediaPackage python deps with dh-virtualenv
integration/zuul : debian-precise-venvPackage python deps with dh-virtualenv
integration/zuul : upstream-debian-sid(WIP) Bump to 2.0.0
integration/config : masterExperimental integration-zuul-debian-glue job

Related Objects

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 1:18 AM
bzimport set Reference to bz46552.
bzimport added a subscriber: Unknown Object (MLST).

This is an upstream issue. A first draft has been published at https://github.com/pabelanger/zuul-deb

Paul Belanger (from OpenStack) worked on the package again this week. I have submitted a bunch of minor amendments https://github.com/pabelanger/zuul-deb/pull/3

Overall the package seems to be very close to be complete.

I have refreshed the package to bring Zuul 1.3.0, the last version before the Gearman feature: https://github.com/pabelanger/zuul-deb/commit/af0293b

We now need a new version to be cut which I requested to upstream with https://bugs.launchpad.net/zuul/+bug/1246799 . We can still install from source :-]

There is little point for Wikimedia to have Zuul deployed via Debian packages. We might want to migrate to git-deploy though.

hashar reopened this task as Open.Feb 10 2015, 3:36 PM

We now rely on a the zuul-cloner client provided by Zuul. It is a pity to have it installed on each of the slaves and a package would definitely be needed.

We need the packaging work to be done for Continuous-Integration-Scaling

hashar added a comment.EditedFeb 10 2015, 3:40 PM

Some of the requirements will most probably conflicts with the packages available for Trusty on apt.wikimedia.org :-/ See task details.

hashar updated the task description. (Show Details)Feb 10 2015, 4:11 PM
hashar set Security to None.

I have poked the internal OPS list about creating Debian packages for python software that have conflicting or missing dependencies. Pasted at P284 follow up on OPS list.

hashar updated the task description. (Show Details)Feb 12 2015, 7:21 PM

Change 191701 had a related patch set uploaded (by Hashar):
Experimental integration-zuul-debian-glue job

https://gerrit.wikimedia.org/r/191701

Patch-For-Review

Change 191701 merged by jenkins-bot:
Experimental integration-zuul-debian-glue job

https://gerrit.wikimedia.org/r/191701

I have a new desktop machine with Debian testing. I have managed to setup an Ubuntu Trusty build image on it using cowbuilder:

root@debianbox# cowbuilder --execute --basepath /packaging/base-trusty.cow/ /bin/cat /etc/lsb-release|grep DIST
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04 LTS"
root@debianbox

I managed to get a rough package locally using a random upstream commit without any of our hack. Good progress so far.

My super lame commands:

Create a cow image for debian sid:

cowbuilder --create --debug \
    --basepath /packaging/base-sid.cow \
    --distribution sid \ 
    --debootstrapopts --variant=buildd \
    --debootstrapopts --verbose

Clone integration/zuul.git and checkout the upstream-debian-sid branch. A few changes needed:

  • Change voluptuous version dependency from <0.7 to >=0.7
  • Add python-pbr (>= 0.5.21)
  • and more....

Then build a package:

# env COWBUILDER_BASE=/packaging  git-buildpackage \
    -nc \
    --git-debian-branch=upstream-debian-sid \
    --git-pbuilder \
    --git-upstream-tree=branch \
    --git-upstream-branch=upstream \
    --git-ignore-new

Change 194399 had a related patch set uploaded (by Hashar):
(WIP) Bump to 2.0.0

https://gerrit.wikimedia.org/r/194399

I am now hitting a wall because dh-virtualenv pip doesn't have network access:

I: dh_virtualenv:65: zuul: Installing dependencies
Downloading/unpacking hacking>=0.9.2,<0.10 (from -r test-requirements.txt (line 1))
  Getting page https://pypi.python.org/simple/hacking
  Could not fetch URL https://pypi.python.org/simple/hacking: <urlopen error [Errno -3] Temporary failure in name resolution>
  Will skip URL https://pypi.python.org/simple/hacking when looking for download links for hacking>=0.9.2,<0.10 (from -r test-requirements.txt (line 1))
  Getting page https://pypi.python.org/simple/
  Could not fetch URL https://pypi.python.org/simple/: <urlopen error [Errno -3] Temporary failure in name resolution>

Change 194399 abandoned by Hashar:
(WIP) Bump to 2.0.0

Reason:
Squashing it in follow up https://gerrit.wikimedia.org/r/#/c/194520/

https://gerrit.wikimedia.org/r/194399

Change 194520 had a related patch set uploaded (by Hashar):
Package python deps with dh-virtualenv

https://gerrit.wikimedia.org/r/194520

Change 195272 had a related patch set uploaded (by Hashar):
Package python deps with dh-virtualenv

https://gerrit.wikimedia.org/r/195272

Change 194520 abandoned by Hashar:
Package python deps with dh-virtualenv

Reason:
Moved to debian/precise-wikimedia branch

https://gerrit.wikimedia.org/r/195272

https://gerrit.wikimedia.org/r/194520

hashar renamed this task from [upstream] Create a Debian package for Zuul to Create a Debian package for Zuul.Mar 9 2015, 9:21 PM

From a mail I sent to the private ops list:

Hello,

To package Zuul [T48552], I gave dh_virtualenv a try.

A new version has been released this morning on my request: [0.9] It comes with an option to reuse system packages, ie ones that have been installed by regular packages.

On integration/zuul.git I have created two new branches:

debian/precise-wikimedia

  • snapshot of upstream as is deployed today
  • fork of upstream debian packaging from 2 years or so ago

patch-queue/debian/precise-wikimedia

  • patches cherry picked patches from upstream maintained using [gbp pq]

Some result: http://people.wikimedia.org/~hashar/debs/zuul/

The /debian/ work is in https://gerrit.wikimedia.org/r/#/c/195272/

I have a couple backporting requests that would ease it, in both cases I am providing the packages from Trusty:

Get dh-virtualenv 0.9 on apt.wikimedia.org:
https://phabricator.wikimedia.org/T91631

Backport python-virtualenv from Trusty to Precise:
https://phabricator.wikimedia.org/T92033

[T48552] https://phabricator.wikimedia.org/T48552
[0.9] https://github.com/spotify/dh-virtualenv/issues/78
[gbp pq] https://honk.sigxcpu.org/piki/development/debian_packages_in_git/

Change 195541 had a related patch set uploaded (by Hashar):
Package python deps with dh-virtualenv

https://gerrit.wikimedia.org/r/195541

I have completed some kind of package for Ubuntu Precise. I know have to test it out and see what happens.

Upstream is moving fast enough that dependencies provided by Ubuntu Precise are mostly obsolete. We should aim at migrating the CI servers to Debian/Jessie.

I have further tweaked the package intended for precise-wikimedia. Patchset 9 of https://gerrit.wikimedia.org/r/#/c/195272/ result is on http://people.wikimedia.org/~hashar/debs/zuul/ . I have installed it on an instance: integration-zuul-packaged.eqiad.wmflabs.

The init scripts defaults are set to prevent zuul and zuul-merger from starting. One has to set START_DAEMON=1.

demon removed a subscriber: demon.Mar 13 2015, 3:20 PM

I need to get the package reviewed by ops.

Change 197328 had a related patch set uploaded (by Hashar):
Package python deps with dh-virtualenv

https://gerrit.wikimedia.org/r/197328

Change 197329 had a related patch set uploaded (by Hashar):
Forward port precise dh-virtualenv to trusty

https://gerrit.wikimedia.org/r/197329

Zuul packaging as been discussed during the weekly ops meeting on 03/16. Andrew B. relayed the info during the RelEng weekly checking and I have apparently a volunteer op. Welcome @fgiunchedi :-|

1/1 with Filippo is scheduled for Tuesday 3/24 9:30am CET.

hashar updated the task description. (Show Details)Mar 24 2015, 8:55 AM

The Precise package has been polished following up the 1/1 with Filippo ( https://gerrit.wikimedia.org/r/195272 ). It installs and zuul-cloner seems to work as expected.

Thanks to all @fgiunchedi reviews, I used patchset 16 of https://gerrit.wikimedia.org/r/#/c/195272/ to deploy zuul as a debian packages on the Precise CI slaves instances we are running in labs (integration-slave100[1-4]).

Just add to:

pip uninstall zuul
pip uninstall zuul  # yeah twice
dpkg -i zuul_2.0.0-304-g685ca22-wmf1precise1_amd64.deb
apt-get install -f  # fix missing dependencies

rm /usr/local/bin/zuul*

I have made Jenkins jobs to point to zuul-cloner instead of /usr/local/bin/zuul-cloner with https://gerrit.wikimedia.org/r/#/c/201451/ .

Monitored it the afternoon, all went fine for the Precise labs slaves \o/

Change 202714 had a related patch set uploaded (by Hashar):
zuul: switch install to a Debian package

https://gerrit.wikimedia.org/r/202714

@fgiunchedi and I have just finished a 1/1. I have further tweaked the debian files for Precise and Trusty, he is going to rebuild both packages and upload them to apt.wikimedia.org . ETA depends on his workload with the statsd migration going on.

Package has been build out of https://gerrit.wikimedia.org/r/#/c/195272/ patchset 19

$ apt-cache policy zuul
zuul:
  Installed: (none)
  Candidate: 2.0.0-304-g685ca22-wmf1precise1
  Version table:
     2.0.0-304-g685ca22-wmf1precise1 0
       1001 http://apt.wikimedia.org/wikimedia/ precise-wikimedia/main amd64 Packages

Change 195272 merged by Hashar:
Package python deps with dh-virtualenv

https://gerrit.wikimedia.org/r/195272

Change 197328 merged by Hashar:
Package python deps with dh-virtualenv

https://gerrit.wikimedia.org/r/197328

The Trusty version has been build out of https://gerrit.wikimedia.org/r/#/c/197329/ patchset 7

apt-cache policy zuul
zuul:
  Installed: 2.0.0-304-g685ca22-wmf1trusty1
  Candidate: 2.0.0-304-g685ca22-wmf1trusty1
  Version table:
 *** 2.0.0-304-g685ca22-wmf1trusty1 0
        100 /var/lib/dpkg/status

Change 197329 merged by Hashar:
Forward port precise dh-virtualenv to trusty

https://gerrit.wikimedia.org/r/197329

Change 202714 merged by Alexandros Kosiaris:
zuul: switch install to a Debian package

https://gerrit.wikimedia.org/r/202714

hashar closed this task as Resolved.Apr 17 2015, 1:08 PM
hashar added a subscriber: akosiaris.

We now have Zuul packages for our Precise and Trusty distributions thanks to @fgiunchedi reviews.

They are deployed on all our production and labs systems. Additional thanks to @akosiaris to have been willing to merge the final puppet patches on a Friday.

Next is to create a package for Jessie but I will track that in another task since CI has no Jessie host yet.

Change 195541 abandoned by Hashar:
Package python deps with dh-virtualenv

Reason:
https://gerrit.wikimedia.org/r/195272 in the debian/precise-wikimedia branch has been merged.

I have rebased the patchqueue on top of it, hence this old dupe change is moot and no more needed.

https://gerrit.wikimedia.org/r/195541

hashar reopened this task as Open.Apr 21 2015, 2:12 PM

Reopening, the package hasn't been provided for Trusty on apt.wikimedia.org.

I have poked @fgiunchedi about the Trusty packages. Rebuild it out of the integration/zuul.git debian/trusty-wikimedia branch.

Result on:

fgiunchedi closed this task as Resolved.Apr 22 2015, 10:53 AM

{{done}}