Page MenuHomePhabricator

CI job debian-glue-non-voting: add support for BACKPORTS=yes
Closed, ResolvedPublic

Description

At the best of my knowledge currently is not possible (correct me if I'm wrong) to set the environment variable BACKPORTS to yes for the debian-glue-non-voting job in the integration/config repository.

My personal use case if for cumin repository, that has a debian branch and is set up to run the debian-glue-non-voting job on every CR on that branch, but it also requires to have BACKPORTS=yes set in the environment for the build to succeed.

Event Timeline

You can inject environmental variables in a job inside the set_parameters function inside parameter_functions in the integration/config repo.

This is where params would go for this job:

https://phabricator.wikimedia.org/diffusion/CICF/browse/master/zuul/parameter_functions.py;aa1f679e2ccabb938a2ce957fc76edb7db3d989e$108

instead of continuing the pattern inside the parameter_functions.py file of hard-coding repository names, i.e.:

if 'debian-glue' in job.name:
    if params['ZUUL_PROJECT'] == 'operations/debs/cumin':
        params['BACKPORTS'] = 'yes'

We could tag the job in zuul/layout.yaml (https://docs.openstack.org/infra/zuul/zuul.html#jobs) and use the tag inside the set_parameters function, i.e.:

parameter_functions.py
if job.tags['debian-glue-backports']:
    params['BACKPORT'] = 'yes'

Although it will be tricky to tag a job for only one repository in zuul/layout.yaml

Thanks @thcipriani for the answers, with my little knowledge of the zuul-jenkins relationship and (in)direct variables settings, it seems to me a fairly normal requirement to be able to configure a repository to run a CI job and specify/set some parameters for it.

In this case the debian-glue/debian-glue-non-voting jobs don't even expose a parameter to export BACKPORT=yes when building the package, but that would be a very quick change to the job to add this feature.
The hardest part seems to be able to set this parameter on a per-repository basis in zuul/layout.yaml as the dependency on backport packages is specific to each repository.

Change 374970 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] debian-glue jobs with BACKPORTS=yes

https://gerrit.wikimedia.org/r/374970

Volans pinged me about it and I went with a basic implementation based on Tyler comment above.

Change 374970 merged by jenkins-bot:
[integration/config@master] debian-glue jobs with BACKPORTS=yes

https://gerrit.wikimedia.org/r/374970

thcipriani assigned this task to hashar.

Volans pinged me about it and I went with a basic implementation based on Tyler comment above.

Now merged.

@Volans next time you have a patch set for the debian branch of cumin, you should see: debian-glue-backports-non-voting run which runs the debian glue job with BACKPORTS='yes'.

Closing this one for now, reopen if not working as expected.

I have done a recheck of the latest change that happened in the debian branch: https://gerrit.wikimedia.org/r/#/c/373513/

https://integration.wikimedia.org/ci/job/debian-glue-backports-non-voting/1/console

00:00:39.892 The following packages have unmet dependencies:
00:00:39.892 pbuilder-satisfydepends-dummy : Depends: debhelper (>= 10) but it is not going to be installed.
00:00:39.892 Depends: dh-python (>= 2.20160609~) but 1.20141111-2 is to be installed.
00:00:39.892 Depends: python-mock (>= 2.0.0) but it is not going to be installed.
00:00:39.892 Depends: python-pytest-runner which is a virtual package.
00:00:39.893 Depends: python-setuptools-scm which is a virtual package.

Though it has BACKPORTS=yes and jessie-wikimedia

The D02backports uses:

00:00:30.551 Get:6 http://apt.wikimedia.org jessie-wikimedia/backports amd64 Packages [77.5 kB]

Ie it adds jessie-wikimedia/backports not jessie/backports!

hashar removed a project: Patch-For-Review.
hashar added a subscriber: akosiaris.

I am not sure what is going on since the hook on the machine looks like:

/var/cache/pbuilder/hooks/jessie/D02backports
#!/bin/sh

# Avoid running hooks if backports is not specified
if [ "${BACKPORTS}" = "yes" ]; then
	cat > /etc/apt/sources.list.d/jessie-backports.list <<-'EOF'
	deb http://mirrors.wikimedia.org/debian jessie-backports main contrib non-free
                                                ^^^^^^^^^^^^^^^^
        ...
	apt-get update
fi

But the D02backports hook does not seem to run apt-get update:

00:00:15.471 Fetched 387 kB in 1s (287 kB/s)
00:00:16.526 Reading package lists...
00:00:16.554 I: user script /srv/pbuilder/build/cow.4788/tmp/hooks/D01apt.wikimedia.org finished

00:00:16.554 I: user script /srv/pbuilder/build/cow.4788/tmp/hooks/D02backports starting
00:00:16.559 I: user script /srv/pbuilder/build/cow.4788/tmp/hooks/D02backports finished

00:00:16.679  -> Attempting to satisfy build-dependencies
00:00:16.816  -> Creating pbuilder-satisfydepends-dummy package

00:00:19.054  python-oslo.utils : Depends: python-monotonic (>= 0.6) which is a virtual package.

I am tempted to say that $BACKPORTS is not exported in the pbuilder script.

Change 380704 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] package_builder: always export $BACKPORTS

https://gerrit.wikimedia.org/r/380704

Change 380704 abandoned by Hashar:
package_builder: always export $BACKPORTS

Reason:
Adjusting the sudo policy to env_keep =BACKPORTS was sufficient

https://gerrit.wikimedia.org/r/380704

So BACKPORTS was shallowed by sudo. I have adjusted the sudo policy in Horizon to env_keep =BACKPORTS. As a result the D02backports hook manage to inject the backports repository and runs apt-get update as expected.

On https://gerrit.wikimedia.org/r/#/c/380706 a recheck produced https://integration.wikimedia.org/ci/job/debian-glue-backports-non-voting/10/console. It eventually fails in override_dh_auto_test because setuptools cant parse the version forged by jenkins-debian-glue:

00:00:55.202    debian/rules override_dh_auto_test
00:00:55.203 make[1]: Entering directory '/tmp/buildd/cumin-1.1.1'
00:00:55.356 SETUPTOOLS_SCM_PRETEND_VERSION="1.1.1-1+0~20170926092642.10+jessie+wikimedia~1.gbp21951a" python setup.py develop --user
00:00:55.649 Traceback (most recent call last):
00:00:55.649   File "setup.py", line 77, in <module>
00:00:55.650     zip_safe=False,
00:00:55.650   File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
00:00:55.650     _setup_distribution = dist = klass(attrs)
00:00:55.650   File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 320, in __init__
00:00:55.650     _Distribution.__init__(self, attrs)
00:00:55.650   File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
00:00:55.651     self.finalize_options()
00:00:55.651   File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 387, in finalize_options
00:00:55.651     ep.load()(self, ep.name, value)
00:00:55.652   File "/usr/lib/python2.7/dist-packages/setuptools_scm/integration.py", line 19, in version_keyword
00:00:55.652     dist.metadata.version = get_version(**value)
00:00:55.652   File "/usr/lib/python2.7/dist-packages/setuptools_scm/__init__.py", line 117, in get_version
00:00:55.653     parsed_version = _do_parse(root, parse)
00:00:55.653   File "/usr/lib/python2.7/dist-packages/setuptools_scm/__init__.py", line 66, in _do_parse
00:00:55.653     return meta(pretended)
00:00:55.654   File "/usr/lib/python2.7/dist-packages/setuptools_scm/version.py", line 86, in meta
00:00:55.654     assert tag is not None, 'cant parse version %s' % tag
00:00:55.654 AssertionError: cant parse version None
00:00:55.664 debian/rules:10: recipe for target 'override_dh_auto_test' failed

At least the packages have been installed which was the goal of this task.