Page MenuHomePhabricator
Create Task
Maniphest T48552

Create a Debian package for Zuul
Closed, ResolvedPublic
Actions

Description

Zuul is deployed form a git repository integration/zuul.git We should make it a proper Debian package.

Upstream bug:

https://bugs.launchpad.net/zuul/+bug/1160076

Requirements from master and whether we have them in WMF Ubuntu Trusty apt repo (as of Feb 10th 2015):

pbr>=0.5.21,<1.0 0.7.0
argparse built in python 2.7
PyYAML>=3.1.0 3.10
Paste 1.7.5.1
WebOb>=1.2.3,<1.3 1.3.1
paramiko>=1.8.0 1.10.1
GitPython>=0.3.2.1 0.3.2~RC1
ordereddictbuilt in python 2.7
python-daemon>=2.0.4 Not met 1.5.5
extras 0.0.3
statsd>=1.0.0,<3.0 Not met 3.0.1
voluptuous>=0.7 Not met 0.8.2
gear>=0.5.4,<1.0.0 Not met 0.4.0 (met in Precise)
apscheduler>=2.1.1,<3.0 2.1.2
PrettyTable>=0.6,<0.8 0.7.2
babel>=1.0 1.3
six>=1.6.0 1.9.0 (got backported recently)

Note we have Zuul installed already on Precise but with a lagged out version.

See Also:
https://launchpad.net/bugs/1160076

Details

Reference
bz46552
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Package python deps with dh-virtualenvintegration/zuulpatch-queue/debian/precise-wikimedia+338 -38
zuul: switch install to a Debian packageoperations/puppetproduction+13 -434
Forward port precise dh-virtualenv to trustyintegration/zuuldebian/trusty-wikimedia+16 -4
Package python deps with dh-virtualenvintegration/zuuldebian/trusty-wikimedia+492 -135
Package python deps with dh-virtualenvintegration/zuuldebian/precise-wikimedia+492 -135
Package python deps with dh-virtualenvintegration/zuuldebian-precise-venv+103 -34
(WIP) Bump to 2.0.0integration/zuulupstream-debian-sid+26 -4
Experimental integration-zuul-debian-glue jobintegration/configmaster+10 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 22 2014, 1:18 AM
• bzimport added projects: Continuous-Integration-Infrastructure, Upstream.
• bzimport set Reference to bz46552.
• bzimport added a subscriber: Unknown Object (MLST).
hashar created this task.Mar 25 2013, 10:36 PM
hashar added a comment.Mar 26 2013, 12:20 PM

This is an upstream issue. A first draft has been published at https://github.com/pabelanger/zuul-deb

hashar added a comment.May 15 2013, 8:25 PM

Paul Belanger (from OpenStack) worked on the package again this week. I have submitted a bunch of minor amendments https://github.com/pabelanger/zuul-deb/pull/3

Overall the package seems to be very close to be complete.

hashar added a comment.Nov 4 2013, 9:04 AM

I have refreshed the package to bring Zuul 1.3.0, the last version before the Gearman feature: https://github.com/pabelanger/zuul-deb/commit/af0293b

We now need a new version to be cut which I requested to upstream with https://bugs.launchpad.net/zuul/+bug/1246799 . We can still install from source :-]

hashar added a comment.Apr 2 2014, 1:04 PM

There is little point for Wikimedia to have Zuul deployed via Debian packages. We might want to migrate to git-deploy though.

hashar reopened this task as Open.Feb 10 2015, 3:36 PM
hashar added a project: Continuous-Integration-Scaling.

We now rely on a the zuul-cloner client provided by Zuul. It is a pity to have it installed on each of the slaves and a package would definitely be needed.

We need the packaging work to be done for Continuous-Integration-Scaling

hashar added a comment.EditedFeb 10 2015, 3:40 PM

Some of the requirements will most probably conflicts with the packages available for Trusty on apt.wikimedia.org :-/ See task details.

hashar added a project: acl*sre-team.Feb 10 2015, 3:41 PM
hashar updated the task description. (Show Details)Feb 10 2015, 4:11 PM
hashar set Security to None.
hashar added a comment.Feb 11 2015, 9:22 PM

I have poked the internal OPS list about creating Debian packages for python software that have conflicting or missing dependencies. Pasted at P284 follow up on OPS list.

hashar mentioned this in P284 [Ops] python packaging and dependencies hell.Feb 11 2015, 9:25 PM
hashar updated the task description. (Show Details)Feb 12 2015, 7:21 PM
hashar moved this task from Backlog to In-progress on the Continuous-Integration-Scaling board.Feb 19 2015, 9:02 PM
gerritbot subscribed.Feb 19 2015, 9:03 PM

Change 191701 had a related patch set uploaded (by Hashar):
Experimental integration-zuul-debian-glue job

https://gerrit.wikimedia.org/r/191701

Patch-For-Review

gerritbot added a comment.Feb 19 2015, 9:10 PM

Change 191701 merged by jenkins-bot:
Experimental integration-zuul-debian-glue job

https://gerrit.wikimedia.org/r/191701

hashar mentioned this in rCICF406f8d9655ba: Experimental integration-zuul-debian-glue job.Feb 19 2015, 9:10 PM
Krinkle moved this task from Untriaged to Next on the Continuous-Integration-Infrastructure board.Mar 2 2015, 2:50 PM
Liuxinyu970226 subscribed.Mar 2 2015, 2:54 PM
hashar added a comment.Mar 3 2015, 1:57 PM

I have a new desktop machine with Debian testing. I have managed to setup an Ubuntu Trusty build image on it using cowbuilder:

root@debianbox# cowbuilder --execute --basepath /packaging/base-trusty.cow/ /bin/cat /etc/lsb-release|grep DIST
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04 LTS"
root@debianbox
hashar added a comment.Mar 4 2015, 8:49 PM

I managed to get a rough package locally using a random upstream commit without any of our hack. Good progress so far.

hashar claimed this task.Mar 4 2015, 8:50 PM
hashar moved this task from Next to In-progress on the Continuous-Integration-Infrastructure board.
hashar added a comment.Mar 4 2015, 9:24 PM

My super lame commands:

Create a cow image for debian sid:

cowbuilder --create --debug \
    --basepath /packaging/base-sid.cow \
    --distribution sid \ 
    --debootstrapopts --variant=buildd \
    --debootstrapopts --verbose

Clone integration/zuul.git and checkout the upstream-debian-sid branch. A few changes needed:

  • Change voluptuous version dependency from <0.7 to >=0.7
  • Add python-pbr (>= 0.5.21)
  • and more....

Then build a package:

# env COWBUILDER_BASE=/packaging  git-buildpackage \
    -nc \
    --git-debian-branch=upstream-debian-sid \
    --git-pbuilder \
    --git-upstream-tree=branch \
    --git-upstream-branch=upstream \
    --git-ignore-new
gerritbot added a comment.Mar 4 2015, 10:10 PM

Change 194399 had a related patch set uploaded (by Hashar):
(WIP) Bump to 2.0.0

https://gerrit.wikimedia.org/r/194399

gerritbot added a project: Patch-For-Review.Mar 4 2015, 10:10 PM
hashar mentioned this in T91631: Provide dh-virtualenv 0.9 package on apt.wikimedia.org Precise and Trusty distributions.Mar 5 2015, 11:09 AM
hashar added a comment.Mar 5 2015, 2:57 PM

I am now hitting a wall because dh-virtualenv pip doesn't have network access:

I: dh_virtualenv:65: zuul: Installing dependencies
Downloading/unpacking hacking>=0.9.2,<0.10 (from -r test-requirements.txt (line 1))
  Getting page https://pypi.python.org/simple/hacking
  Could not fetch URL https://pypi.python.org/simple/hacking: <urlopen error [Errno -3] Temporary failure in name resolution>
  Will skip URL https://pypi.python.org/simple/hacking when looking for download links for hacking>=0.9.2,<0.10 (from -r test-requirements.txt (line 1))
  Getting page https://pypi.python.org/simple/
  Could not fetch URL https://pypi.python.org/simple/: <urlopen error [Errno -3] Temporary failure in name resolution>
gerritbot added a comment.Mar 9 2015, 10:39 AM

Change 194399 abandoned by Hashar:
(WIP) Bump to 2.0.0

Reason:
Squashing it in follow up https://gerrit.wikimedia.org/r/#/c/194520/

https://gerrit.wikimedia.org/r/194399

gerritbot added a comment.Mar 9 2015, 10:41 AM

Change 194520 had a related patch set uploaded (by Hashar):
Package python deps with dh-virtualenv

https://gerrit.wikimedia.org/r/194520

gerritbot added a comment.Mar 9 2015, 3:38 PM

Change 195272 had a related patch set uploaded (by Hashar):
Package python deps with dh-virtualenv

https://gerrit.wikimedia.org/r/195272

gerritbot added a comment.Mar 9 2015, 3:38 PM

Change 194520 abandoned by Hashar:
Package python deps with dh-virtualenv

Reason:
Moved to debian/precise-wikimedia branch

https://gerrit.wikimedia.org/r/195272

https://gerrit.wikimedia.org/r/194520

hashar renamed this task from [upstream] Create a Debian package for Zuul to Create a Debian package for Zuul.Mar 9 2015, 9:21 PM
hashar added a subtask: T91631: Provide dh-virtualenv 0.9 package on apt.wikimedia.org Precise and Trusty distributions.
hashar added a comment.Mar 9 2015, 10:34 PM

From a mail I sent to the private ops list:

Hello,

To package Zuul [T48552], I gave dh_virtualenv a try.

A new version has been released this morning on my request: [0.9] It comes with an option to reuse system packages, ie ones that have been installed by regular packages.

On integration/zuul.git I have created two new branches:

debian/precise-wikimedia

  • snapshot of upstream as is deployed today
  • fork of upstream debian packaging from 2 years or so ago

patch-queue/debian/precise-wikimedia

  • patches cherry picked patches from upstream maintained using [gbp pq]

Some result: http://people.wikimedia.org/~hashar/debs/zuul/

The /debian/ work is in https://gerrit.wikimedia.org/r/#/c/195272/

I have a couple backporting requests that would ease it, in both cases I am providing the packages from Trusty:

Get dh-virtualenv 0.9 on apt.wikimedia.org:
https://phabricator.wikimedia.org/T91631

Backport python-virtualenv from Trusty to Precise:
https://phabricator.wikimedia.org/T92033

[T48552] https://phabricator.wikimedia.org/T48552
[0.9] https://github.com/spotify/dh-virtualenv/issues/78
[gbp pq] https://honk.sigxcpu.org/piki/development/debian_packages_in_git/

gerritbot added a comment.Mar 10 2015, 11:59 AM

Change 195541 had a related patch set uploaded (by Hashar):
Package python deps with dh-virtualenv

https://gerrit.wikimedia.org/r/195541

hashar added a comment.Mar 10 2015, 10:21 PM

I have completed some kind of package for Ubuntu Precise. I know have to test it out and see what happens.

Upstream is moving fast enough that dependencies provided by Ubuntu Precise are mostly obsolete. We should aim at migrating the CI servers to Debian/Jessie.

hashar added a comment.Mar 13 2015, 3:19 PM

I have further tweaked the package intended for precise-wikimedia. Patchset 9 of https://gerrit.wikimedia.org/r/#/c/195272/ result is on http://people.wikimedia.org/~hashar/debs/zuul/ . I have installed it on an instance: integration-zuul-packaged.eqiad.wmflabs.

The init scripts defaults are set to prevent zuul and zuul-merger from starting. One has to set START_DAEMON=1.

• demon unsubscribed.Mar 13 2015, 3:20 PM
hashar added a project: Blocked-on-Operations.Mar 13 2015, 3:20 PM

I need to get the package reviewed by ops.

hashar added a subtask: T92684: Get python-gear 0.5.5 to trusty-wikimedia and jessie-wikimedia.Mar 13 2015, 9:50 PM
gerritbot added a comment.Mar 17 2015, 3:20 PM

Change 197328 had a related patch set uploaded (by Hashar):
Package python deps with dh-virtualenv

https://gerrit.wikimedia.org/r/197328

gerritbot added a comment.Mar 17 2015, 3:20 PM

Change 197329 had a related patch set uploaded (by Hashar):
Forward port precise dh-virtualenv to trusty

https://gerrit.wikimedia.org/r/197329

hashar added a subscriber: fgiunchedi.Mar 17 2015, 9:16 PM

Zuul packaging as been discussed during the weekly ops meeting on 03/16. Andrew B. relayed the info during the RelEng weekly checking and I have apparently a volunteer op. Welcome @fgiunchedi :-|

hashar added a comment.Mar 19 2015, 10:45 AM

1/1 with Filippo is scheduled for Tuesday 3/24 9:30am CET.

akosiaris closed subtask T92033: Backport python-virtualenv 1.11.4 from Trusty to Precise as Resolved.Mar 19 2015, 3:07 PM
hashar mentioned this in T84956: Create basic puppet role for Sentry.Mar 21 2015, 8:39 PM
hashar updated the task description. (Show Details)Mar 24 2015, 8:55 AM
akosiaris closed subtask T91631: Provide dh-virtualenv 0.9 package on apt.wikimedia.org Precise and Trusty distributions as Resolved.Mar 24 2015, 12:55 PM
hashar added a comment.Mar 24 2015, 1:53 PM

The Precise package has been polished following up the 1/1 with Filippo ( https://gerrit.wikimedia.org/r/195272 ). It installs and zuul-cloner seems to work as expected.

hashar mentioned this in T94409: Upgrade Zuul server to latest upstream.Mar 30 2015, 2:18 PM
hashar added a parent task: T94409: Upgrade Zuul server to latest upstream.
fgiunchedi closed subtask T92684: Get python-gear 0.5.5 to trusty-wikimedia and jessie-wikimedia as Resolved.Apr 2 2015, 10:42 AM
hashar added a comment.Apr 2 2015, 3:15 PM

Thanks to all @fgiunchedi reviews, I used patchset 16 of https://gerrit.wikimedia.org/r/#/c/195272/ to deploy zuul as a debian packages on the Precise CI slaves instances we are running in labs (integration-slave100[1-4]).

Just add to:

pip uninstall zuul
pip uninstall zuul  # yeah twice
dpkg -i zuul_2.0.0-304-g685ca22-wmf1precise1_amd64.deb
apt-get install -f  # fix missing dependencies

rm /usr/local/bin/zuul*

I have made Jenkins jobs to point to zuul-cloner instead of /usr/local/bin/zuul-cloner with https://gerrit.wikimedia.org/r/#/c/201451/ .

Monitored it the afternoon, all went fine for the Precise labs slaves \o/

gerritbot added a comment.Apr 8 2015, 12:13 PM

Change 202714 had a related patch set uploaded (by Hashar):
zuul: switch install to a Debian package

https://gerrit.wikimedia.org/r/202714

hashar added a parent task: T61991: Zuul: python git assert error assert len(fetch_info_lines) == len(fetch_head_info).Apr 13 2015, 10:03 AM
hashar added a project: Zuul.Apr 16 2015, 9:38 AM

@fgiunchedi and I have just finished a 1/1. I have further tweaked the debian files for Precise and Trusty, he is going to rebuild both packages and upload them to apt.wikimedia.org . ETA depends on his workload with the statsd migration going on.

hashar added a comment.Apr 16 2015, 1:53 PM

Package has been build out of https://gerrit.wikimedia.org/r/#/c/195272/ patchset 19

$ apt-cache policy zuul
zuul:
  Installed: (none)
  Candidate: 2.0.0-304-g685ca22-wmf1precise1
  Version table:
     2.0.0-304-g685ca22-wmf1precise1 0
       1001 http://apt.wikimedia.org/wikimedia/ precise-wikimedia/main amd64 Packages
gerritbot added a comment.Apr 16 2015, 1:59 PM

Change 195272 merged by Hashar:
Package python deps with dh-virtualenv

https://gerrit.wikimedia.org/r/195272

gerritbot added a comment.Apr 16 2015, 2:05 PM

Change 197328 merged by Hashar:
Package python deps with dh-virtualenv

https://gerrit.wikimedia.org/r/197328

hashar added a comment.Apr 16 2015, 2:06 PM

The Trusty version has been build out of https://gerrit.wikimedia.org/r/#/c/197329/ patchset 7

apt-cache policy zuul
zuul:
  Installed: 2.0.0-304-g685ca22-wmf1trusty1
  Candidate: 2.0.0-304-g685ca22-wmf1trusty1
  Version table:
 *** 2.0.0-304-g685ca22-wmf1trusty1 0
        100 /var/lib/dpkg/status
gerritbot added a comment.Apr 16 2015, 2:06 PM

Change 197329 merged by Hashar:
Forward port precise dh-virtualenv to trusty

https://gerrit.wikimedia.org/r/197329

hashar mentioned this in rCIZU4aff9e360fad: Forward port precise dh-virtualenv to trusty.Apr 16 2015, 2:51 PM
hashar mentioned this in rCIZU6de26ca279bb: Package python deps with dh-virtualenv.
gerritbot added a comment.Apr 17 2015, 12:10 PM

Change 202714 merged by Alexandros Kosiaris:
zuul: switch install to a Debian package

https://gerrit.wikimedia.org/r/202714

hashar mentioned this in rOPUP0889192319ee: zuul: switch install to a Debian package.Apr 17 2015, 12:10 PM
hashar closed this task as Resolved.Apr 17 2015, 1:08 PM
hashar added a subscriber: akosiaris.

We now have Zuul packages for our Precise and Trusty distributions thanks to @fgiunchedi reviews.

They are deployed on all our production and labs systems. Additional thanks to @akosiaris to have been willing to merge the final puppet patches on a Friday.

Next is to create a package for Jessie but I will track that in another task since CI has no Jessie host yet.

gerritbot added a comment.Apr 17 2015, 1:43 PM

Change 195541 abandoned by Hashar:
Package python deps with dh-virtualenv

Reason:
https://gerrit.wikimedia.org/r/195272 in the debian/precise-wikimedia branch has been merged.

I have rebased the patchqueue on top of it, hence this old dupe change is moot and no more needed.

https://gerrit.wikimedia.org/r/195541

hashar moved this task from In-progress to Done on the Continuous-Integration-Scaling board.Apr 17 2015, 9:00 PM
Liuxinyu970226 unsubscribed.Apr 18 2015, 1:26 AM
hashar mentioned this in T96463: Move integration-zuul-layout* jobs to labs slaves.Apr 20 2015, 1:34 PM
hashar mentioned this in rCICF2ecf04a649f8: Move integration-zuul-layout* to labs.Apr 20 2015, 1:54 PM
hashar reopened this task as Open.Apr 21 2015, 2:12 PM

Reopening, the package hasn't been provided for Trusty on apt.wikimedia.org.

hashar added a comment.Apr 22 2015, 9:37 AM

I have poked @fgiunchedi about the Trusty packages. Rebuild it out of the integration/zuul.git debian/trusty-wikimedia branch.

Result on:

fgiunchedi closed this task as Resolved.Apr 22 2015, 10:53 AM

{{done}}

hashar moved this task from In-progress to Done on the Continuous-Integration-Infrastructure board.Apr 22 2015, 2:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits