context

For repositories having a Debian package, we run the Jenkins job debian-glue. It is a wrapper around git buildpackage/cowbuilder and get the target distribution from the debian/changelog file. Hence it builds for a single distribution.

The job only triggers when a file under debian/ directory is altered.

The CI configuration for labs/toollabs looks like:

- name: labs/toollabs
  template:
    - name: tox-docker
    - name: php55lint
    - name: php70lint
  test:
   - debian-glue
  gate-and-submit:
   - debian-glue

tox-docker runs tox under Debian Stretch with python 2.7, 3.4, 3.5, 3.6, 3.7 available. It always trigger.

fix up

For multiple distributions, we need a new set of debian-glue jobs that come with an hardcoded distribution instead of guessing it from the debian/changelog.

Being a native package, ideally the jobs should always triggered, even when no files is changed in debian/ directory. Though probably it is easy enough to just add an entry to debian/changelog whenever doing a patch. It is not ideal, but simpler CI wise though jobs would NOT be triggered if someone forget to bump the changelog. That can be figured out later though.

The idea comes from https://gerrit.wikimedia.org/r/#/c/labs/toollabs/+/476092/ which fails on CI. That is due to the changelog pointing at unstable and python 3.7 got introduced at some point and thus breaking the build.

I am fine targeting unstable. Since the master branch is intended to be for multiple distributions, we surely need to build the labs/toollabs Debian package for all the distributions it supports.

Change 481670 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] debian-glue jobs with hardcoded distributions

https://gerrit.wikimedia.org/r/481670

Mentioned in SAL (#wikimedia-releng) [2019-01-02T10:19:32Z] <hashar> updating all debian-glue jobs and creating new ones with hardcoded distributions (trusty, jessie, stretch, unstable) T210780

Change 481670 merged by jenkins-bot:
[integration/config@master] debian-glue jobs with hardcoded dists for labs/toollabs

https://gerrit.wikimedia.org/r/481670

Change 481835 had a related patch set uploaded (by Hashar; owner: Hashar):
[labs/toollabs@master] Dummy build to exercise Wikimeda CI

https://gerrit.wikimedia.org/r/481835

I have send a dummy change in Gerrit that just bump the changelog (#481835):

The failures on Trusty and Jessie are due to lintian. Test results:

• https://integration.wikimedia.org/ci/job/debian-glue-trusty-non-voting/1/testReport/
  • E: toollabs changes: bad-distribution-in-changes-file unstable
    • Some how CI creates a dummy changelog, I guess it does not recognizes trusty and fallback to unstable :/
  • E: toollabs source: missing-build-dependency-for-dh-addon python3 => python3 | python3-all | python3-dev | python3-all-dev
    • The package has a build dependency set to python3:any which apparently is not recognized by the lintian version in Trusty?

• https://integration.wikimedia.org/ci/job/debian-glue-jessie-non-voting/1/testReport/
  • E: toollabs source: missing-build-dependency-for-dh-addon python3 => python3 | python3-all | python3-dev | python3-all-dev
    • The package has a build dependency set to python3:any which apparently is not recognized by the lintian version in Jessie?

For Trusty the dsc has:

Changes: 
 toollabs (1.35+nmu1~20190102102533.1+trusty) unstable; urgency=medium
 .
   [ Antoine Musso ]
   * Non-maintainer upload.
   * Dummy build for CI - T210780
 .
   [ jenkins-debian-glue Autobuilder ]
   * SNAPSHOT autobuild for unreleased 1.35+nmu1 via jenkins-debian-glue.

In T210780#4848466, @hashar wrote:

• https://integration.wikimedia.org/ci/job/debian-glue-trusty-non-voting/1/testReport/
  • E: toollabs changes: bad-distribution-in-changes-file unstable
    • Some how CI creates a dummy changelog, I guess it does not recognizes trusty and fallback to unstable :/

The d/changelog file actually has unstable as distribution. Also, the lintian warning refers to the .changes file, generated after the package build

Not sure about git buildpackage/cowbuilder but with sbuild you can specify for which distro you want to build the package independent of what d/changelog says (the -d option).
That will set the Distribution field in the .changes file to what you want.
There is another way to workaround this, and is by using lintian profiles:

It is possible that you are uploading for a different distribution than the one Lintian is checking for. In that case, passing --profile $VENDOR may fix this warning.

Not sure how much work on your side could mean any of the solutions.

• E: toollabs source: missing-build-dependency-for-dh-addon python3 => python3 | python3-all | python3-dev | python3-all-dev
  • The package has a build dependency set to python3:any which apparently is not recognized by the lintian version in Trusty?

This usually means the package requires a build-dep on dh-python. I don't think this error is induced by the CI system.

• https://integration.wikimedia.org/ci/job/debian-glue-jessie-non-voting/1/testReport/
  • E: toollabs source: missing-build-dependency-for-dh-addon python3 => python3 | python3-all | python3-dev | python3-all-dev
    • The package has a build dependency set to python3:any which apparently is not recognized by the lintian version in Jessie?

Same. This usually means the package requires a build-dep on dh-python. I don't think this error is induced by the CI system.

Not sure about git buildpackage/cowbuilder but with sbuild you can specify for which distro you want to build the package independent of what d/changelog says (the -d option).
That will set the Distribution field in the .changes file to what you want.

The CI jobs use the cow images set by the puppet module package_buider wich uses pbuilder/cowbuilder. There is a wrapping shell script Jenkins Debian Glue which uses git buildpackage on top of that :)

I eventually found the issue which is that the Jenkins Debian Glue script generate-git-snapshot does not set the distribution when creating the source changes (THANKS for the hint!). The DIST environment variable has to be set which is then passed as git buildpackage --distribution=$DIST.

I have set DIST and the .changes is correct. A side effect is the source changes is kept around (as <foo>_source.changes).

I have confirmed the Distribution field is properly set. Eg jessie-wikimedia or trusty. Thus the lintian error E: toollabs changes: bad-distribution-in-changes-file unstable is gone!

Change 481870 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] debian-glue fix Distribution in .changes file

https://gerrit.wikimedia.org/r/481870

Change 481870 merged by jenkins-bot:
[integration/config@master] debian-glue fix Distribution in .changes file

https://gerrit.wikimedia.org/r/481870

There is another way to workaround this, and is by using lintian profiles:

It is possible that you are uploading for a different distribution than the one Lintian is checking for. In that case, passing --profile $VENDOR may fix this warning.

We do not use lintian profiles per see. lintian is just run inside the same cowbuilder to use the version from the target distribution.

For Jessie and Trusty lintian causes:

E: toollabs source: missing-build-dependency-for-dh-addon python3 => python3 | python3-all | python3-dev | python3-all-dev

I did a bit of digging, labs/toollabs has Build-Depends: python3:any and dh-python in Jessie/Stretch depends on python3-minimal. It looks like python3:any does not recognizes python3-minimal.

Eventually I have found https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=803164 lintian: Bogus missing-python-build-dependency and missing-build-dependency-for-dh-addon which should be fixed in Lintian 2.5.44 (Jessie has 2.5.30).

• data/debhelper/dh_addons-manual: + [NT] Add ":any" to the dependencies for the python add-ons. Thanks to Barry Warsaw for reporting the bug. (Closes: #803164)

That leads to lintian commit 8d53737 - Add missing ":any" to some python relations.

We might be able to request the Jessie version to be fixed, though that is probably unlikely to happen. Maybe we can instead add a lintian override?

Change 481876 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] labs/toollabs .deb works for Stretch

https://gerrit.wikimedia.org/r/481876

Change 481876 merged by jenkins-bot:
[integration/config@master] labs/toollabs .deb works for Stretch

https://gerrit.wikimedia.org/r/481876

Change 484772 had a related patch set uploaded (by Hashar; owner: Hashar):
[labs/toollabs@master] d/control: Build-Depends source on python3

https://gerrit.wikimedia.org/r/484772

Changing Build-Depends from python3:any to python3 fixes it. https://gerrit.wikimedia.org/r/#/c/labs/toollabs/+/484772/ reports:

debian-glue-unstable SUCCESS in 40s
debian-glue-trusty-non-voting SUCCESS in 37s (non-voting)
debian-glue-jessie-non-voting SUCCESS in 56s (non-voting)
debian-glue-stretch-non-voting SUCCESS in 33s (non-voting)

Once that change is merged, I will make all those CI jobs voting.

Change 481835 abandoned by Hashar:
Dummy build to exercise Wikimeda CI

https://gerrit.wikimedia.org/r/481835

Change 486263 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] labs/toollabs debian job for trusty/jessie pass

https://gerrit.wikimedia.org/r/486263

Change 484772 merged by jenkins-bot:
[labs/toollabs@master] d/control: Build-Depends source on python3

https://gerrit.wikimedia.org/r/484772

Change 486263 merged by jenkins-bot:
[integration/config@master] labs/toollabs debian job for trusty/jessie pass

https://gerrit.wikimedia.org/r/486263