Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Jul 20 2016, 1:35 PM

Description

I would like to get Zuul upgraded on scandium.eqiad.wmnet which is a Jessie machine and hosting the zuul-merger daemon.

That will cause a short pause of CI processing while the zuul-merger is upgraded (a couple minutes)

The maintenance window would be around half an hour.

Package

I have prepared a new package and tested it out on lab. It seems fine.

The package is on:
https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-391-gbc58ea3-jessie/

Once happy, we would want to publish the package on http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/thirdparty

Upgrade

ssh scandium.eqiad.wmnet
service zuul-merger stop
wget https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-391-gbc58ea3-jessie/zuul_2.1.0-391-gbc58ea3-wmf1jessie1_amd64.deb
dpkg -i zuul_2.1.0-391-gbc58ea3-wmf1jessie1_amd64.deb
service zuul-merger start

Then watch /var/log/zuul/merger.log and on the zuul server (gallium) what for /var/log/zuul/zuul.log

Rollback

service zuul-merger stop
Reinstall Zuul using the version from apt.wikimedia.org 2.1.0-60-g1cc37f7-wmf4jessie1.
service zuul-merger start

Watch /var/log/zuul/merger.log and on the zuul server (gallium) what for /var/log/zuul/zuul.log

Related Objects

Mentioned In: T145057: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger)
Mentioned Here: T145057: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger)
T128569: Zuul deadlocks if unknown repo has activity in Gerrit
T93812: Change force merged cause a deadlock in Zuul gate-and-submit pipeline

Event Timeline

hashar created this task.Jul 20 2016, 1:35 PM

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 20 2016, 1:35 PM

Danny_B added a project: Zuul.Jul 20 2016, 2:50 PM

Paladox subscribed.Jul 20 2016, 2:52 PM

Let's do this tomorrow morning maybe?

In T140894#2482428, @demon wrote:

Let's do this tomorrow morning maybe?

@elukey proposed to review the package and we had a quick discussion about it. Turns out upgrading the zuul merger on a friday evening is not a good idea, so we will look at it monday ;-}

hashar added a subscriber: fgiunchedi.Jul 22 2016, 2:52 PM

Since I have filled this task, the zuul server on gallium (Precise) has been upgraded to further catchup with upstream ( zuul_2.1.0-391-gbc58ea3 ).

I have merged all changes for jessie and rebuild the package. It looks good and is at https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-391-gbc58ea3-jessie/zuul_2.1.0-391-gbc58ea3-wmf1jessie1_amd64.deb

Mentioned in SAL [2016-07-28T12:18:49Z] <hashar> installed 2.1.0-391-gbc58ea3-wmf1jessie1 on zuul-dev-jessie.integration.eqiad.wmflabs T140894

Mentioned in SAL [2016-07-28T12:51:09Z] <elukey> upgrading zuul-merger to zuul_2.1.0-391-gbc58ea3-wmf1jessie (T140894)

Luca has done the upgrade and it went flawlessly. I have double checked a few use cases (such as depending jobs, operations/puppet) and it works all fine.

We will want to push https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-391-gbc58ea3-jessie/ to apt.wm.o jessie-wikimedia/thirdparty

I had to rebuild the package on Precise for T93812 and I have rebuild the Jessie one as well. The patch adjust some sleep and is tiny https://phabricator.wikimedia.org/rCIZUe094fd8a68665cb82f34ecf9b558607d2c27a016

Have to upgrade scandium from wmf1 to wmf2 and put the wmf2 package on apt.wm.o. Sorry :(

Package material is at:

https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-391-gbc58ea3-jessie/

.deb https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-391-gbc58ea3-jessie/zuul_2.1.0-391-gbc58ea3-wmf2jessie1_amd64.deb

Then we will be properly in sync and there is no more upgrade / patch I envision to do in the short term :]

[12:43:25] <@elukey> !log upgrading zuul-merger to zuul_2.1.0-391-gbc58ea3-wmf2jessie1_amd64.deb on scandium

\o/

Good. Will have to push the packages to apt.wm.o:

Files are in https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-391-gbc58ea3-jessie/ and that should land under jessie-wikimedia/thirdparty

And for Precise files are in https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-391-gbc58ea3/ for precise-wikimedia/thirdparty.

@hashar there is a problem uploading the jessie package:

Unable to find pool/thirdparty/z/zuul/zuul_2.1.0-391-gbc58ea3.orig.tar.gz needed by zuul_2.1.0-391-gbc58ea3-wmf2jessie1.dsc!
Perhaps you forgot to give dpkg-buildpackage the -sa option,
 or you could try --ignore=missingfile to guess possible files to use.
Deleting files just added to the pool but not used.
(to avoid use --keepunusednewfiles next time)
There have been errors!

The .changes file does not contain the .origin.tar.gz reference so reprepro tries to find it in pool/thirdparty. I think that we'd need to rebuild the package!

Looks like I have been building it without dpkg-gen-changes -sa to force the inclusion of the original tarball in the .changes file.

It is listed in the .dsc at least:

Checksums-Sha1:
 0e33d993186f71d31fe0f445976804271feb626c 207396 zuul_2.1.0-391-gbc58ea3.orig.tar.gz
Checksums-Sha256:
 ed2722969e07d083415a8850144c065e4f94749987959bd0a7980b0501bf31dd 207396 zuul_2.1.0-391-gbc58ea3.orig.tar.gz
Files:
 1a0ee1e204a50a1ca36d2bf1473ea2e6 207396 zuul_2.1.0-391-gbc58ea3.orig.tar.gz

The corresponding tarball is: https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-391-gbc58ea3-jessie/zuul_2.1.0-391-gbc58ea3.orig.tar.gz

Can we manually alter zuul_2.1.0-391-gbc58ea3-wmf2jessie1_amd64.changes to add the missing fields? Or manually copy zuul_2.1.0-391-gbc58ea3.orig.tar.gz to pool/thirdparty/z/zuul/zuul_2.1.0-391-gbc58ea3.orig.tar.gz ?

One of the reason is that I don't have access to my build machine right now, the other is the Zuul packaging download dependencies from pypi which are not necessarily stuck to a given version. So a repackaging might yield new versions which will have to be carefully checked / tested etc. A bit of a burden I would rather skip if we can manually hack the .changes or copy the file directly.

I will rebuild Zuul to try a patch for T128569 ( https://github.com/openstack-infra/zuul/commit/a8b90b38094587bbd82ffa5e4028aef1dfd02987 ). So there is no need to push the zuul_2.1.0-391-gbc58ea3-wmf2jessie1_amd64 one, it will be obsolete pretty soon :)

@hashar we should also bump the version to 2.5.0 per https://github.com/openstack-infra/zuul/releases/tag/2.5.0

Also we should do the same for precise and trusty please?

I have bumped the version to zuul_2.5.0-8-gcbc7f62-wmf2jessie1 and the new package .changes file properly list the orig tarball. Task is T145057

Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger)Closed, ResolvedPublicActions