Page MenuHomePhabricator

Upgrade jenkins-debian-glue to v0.20.0
Closed, ResolvedPublic

Description

We run a fork of v0.18.4 and need the version bump for T212773 and v0.20.0 for T186494. Last releases:

v0.19.0Nov 2018
v0.20.0Jan 2019

Changelogs:

v0.20.0

jenkins-debian-glue (0.20.0) unstable; urgency=medium

  [ Victor Seva ]
  * [e2beae5] lintian-junit-report: support processing a
    lintian output file

  [ Rémi Palancher ]
  * [90ecaab] Ability to add label/version in repos conf

  [ John Goerzen ]
  * [d9d9b8f] Allow MIRRORSITE to pass sudo

  [ Antoine Musso ]
  * [e336c9e] Add pbuilder hook to run lintian (inside the
    target system, using the cowbuiler environment)

  [ Kienan Stewart ]
  * [41b7934] Check if reprepro distribution exists
  * [dc5dbf5] Use sudo command when grepping for distribution codename

  [ Laurent Declercq ]
  * [b63ab9d] Typo in NOTES.org
  * [7639edb] Added pbuilder hook to force DPKG unsafe IO (unpack)

  [ Will DeBerry ]
  * [4a5093c] autopkg_binary variable needs to always be set

  [ Mykola Malkov ]
  * [52f3099] Fix 'skip tests' message for perlcritic test

  [ Michael Prokop ]
  * [9ac5ef6] piuparts_wrapper: ensure COMPONENTS variable is unset before
    invoking debootstrap
  * [726137c] build-and-provide-package: unify identition and logging
    message style
  * [0b04226] build-and-provide-package: also run generate-reprepro-
    codename if reprepro configuration is missing
  * [5b8aab8] puppet: ensure we have key available before setting up
    sources.list
  * [71a21fb] puppet: use openjdk-8-jdk-headless on Ubuntu 18.04

 -- Michael Prokop <mika@debian.org>  Tue, 15 Jan 2019 17:49:39 +0100

v0.19.0

jenkins-debian-glue (0.19.0) unstable; urgency=medium

  [ Guillem Jover ]
  * [616d23e] Wrap and sort dependency fields
  * [d9db869] Switch from Priority extra to optional
  * [b9196d5] Remove boilerplate comment from debian/rules
  * [dce1238] Add new debian/.gitignore file
  * [575728b] Bump debhelper compatibility level to 10
  * [cdf1b88] Use the canonical URL for the debian/copyright Format field
  * [00602c6] Remove Copyright notice from License field
  * [2126ad0] Bump Standards-Version to 4.1.1

  [ Michael Prokop ]
  * [3a0a8fa] puppet: ensure recent Jenkins Git client plugin dependencies are fulfilled
  * [3c53ca6] puppet: ensure that apt-transport-https is present
  * [3529ae6] piuparts_tap: don't check only for error code 100 but accept any digit
  * [d27375d] puppet: make check for installed packages more reliable.
    Thanks to Guillem Jover for feedback and code review
  * [e1db6a7] puppet: don't rely on outside variable inside package_check.
    Thanks to Alex Lutay
  * [9e2117c] Support autopkgtest v5. Thanks to Christoph Berg <myon@debian.org> for feedback
  * [e270387] debian: run wrap-and-sort
  * [525467d] build-and-provide-package: explicitly set TERM environment variable in cowbuilder runs
  * [d10af05] Bump Standards-Version to 4.2.1
  * [cab6fbf] Adjust lintian-overrides file for new lintian output behavior

 -- Michael Prokop <mika@debian.org>  Tue, 06 Nov 2018 11:07:08 +0100

Event Timeline

hashar created this task.Jan 2 2019, 11:23 AM
This comment was removed by Aklapper.
hashar renamed this task from Upgrade jenkins-debian-glue to v0.19.0 to Upgrade jenkins-debian-glue to v0.20.0.Jan 17 2019, 8:54 AM
hashar updated the task description. (Show Details)
hashar added a project: Packaging.

Change 484975 had a related patch set uploaded (by Hashar; owner: Legoktm):
[integration/config@master] php-composer-security-docker: Use composer:0.1.8 image

https://gerrit.wikimedia.org/r/484975

My patches are in v0.20.0. I used to have the package build from operations/debs/jenkins-debian-glue which had our fork but it is no more needed.

Instead we can just rebuild upstream version which has landed in Debian testing/unstable https://packages.debian.org/buster/jenkins-debian-glue

Looks like we need:

http://deb.debian.org/debian/pool/main/j/jenkins-debian-glue/jenkins-debian-glue_0.20.0.dsc
http://deb.debian.org/debian/pool/main/j/jenkins-debian-glue/jenkins-debian-glue_0.20.0.tar.xz

Or maybe reprepro can let us import it directly. The binary packages should be added to jessie-wikimedia/main and if at all possible to stretch-wikimedia/main as well.

hashar edited projects, added Operations; removed Patch-For-Review.Jan 21 2019, 4:59 PM

I have managed to build the package for both jessie and stretch without any issues! :)

To clarify from discussions I had:

  • the packages are not installed on production servers, solely on a few WMCS instances used for CI.
  • they are only used for CI needs
  • the source package is a Debian native package, seems to qualify for the main component.

Sounds like Operations can now get the source package from Debian.org, rebuild it for Jessie and Stretch and upload the resulting binaries to jessie-wikimedia/main and stretch-wikimedia/main :]

jijiki triaged this task as Normal priority.Feb 7 2019, 12:46 PM
  • the source package is a Debian native package, seems to qualify for the main component.

We have component/ci for a while, shouldn't it be uploaded there?

jbond added a subscriber: jbond.Mar 11 2019, 11:07 AM

On Jessie we had it in jessie-wikimedia/main, which predates the introduction of component/xxx in our apt repositories. So we have:

$ apt-cache policy jenkins-debian-glue
jenkins-debian-glue:
  Installed: 0.18.4-wmf3
  Candidate: 0.18.4-wmf3
  Version table:
 *** 0.18.4-wmf3 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status

Eventually upstream got the package uploaded and it has made it in Stretch. I guess under stretch/main. So yeah we should align with the new way of doing things and add it to component/ci for both distributions.

The only question I have is: on Stretch would apt default to the latest version or will I need an apt preference to select the version from component/ci? Or in other worlds, are packages from component/xxx taking over or are their optional? ;)

Change 495681 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] Add component/ci wikimedia repository to CI hosts

https://gerrit.wikimedia.org/r/495681

jbond added a comment.Mar 11 2019, 1:19 PM

i have built this and added it to {jessie,stretch}-wikimedia in component/ci. It appears that the CI servers did not have components/ci configured so i have also created a change to add the repo to those servers. As far as i can tell this only needs to go to contint2001.wikimedia.org and contint1001.wikimedia.org, which both run Jessie. Have i missed some nodes which may also need this repo?

Thanks John

@hashar i have taken another look at the patch i created yesterday and i now think it is incorrect. As far as i can tell profile::ci::package_builder which is the only class that attempts to install jenkins-debian-glue is never run on any server. profile::ci::package_builder is included in role::ci::slave::labs however

it/puppet $ grep -lr profile::ci::package_builder ./               
./modules/role/manifests/ci/slave/labs.pp
./modules/profile/manifests/ci/package_builder.pp
git/puppet $
git/puppet $ grep -lr 'role::ci::slave::labs[^:]' ./                    
./modules/role/manifests/ci/slave/labs.pp
git/puppet $
git/puppet $ grep ci::slave::labs manifests/site.pp
git/puppet $

Further to this cumin produces no results

jbond@cumin1001:~$ sudo cumin 'C:profile::ci::package_builder'
No hosts found that matches the query
jbond@cumin1001:~$ sudo cumin 'C:ci::slave::labs'
No hosts found that matches the query
jbond@cumin1001:~$ sudo cumin 'R:Package = jenkins-debian-glue'
No hosts found that matches the query

As such I'm unsure how best to proceed. Which servers need the jenkins-debian-glue package?

Thanks John

> As far as i can tell this only needs to go to contint2001.wikimedia.org and contint1001.wikimedia.org

Those run the CI stack (Jenkins, Zuul etc) but they do not build the packages. Jenkins connects to WMCS instances that then run the actual jobs. Those instances are in the integration WMCS tenant and they are provisioned with puppet classes which are applied in Horizon.

The instances building the debian packages are:

integration-slave-jessie-1001
integration-slave-jessie-1002
integration-slave-jessie-1004

They have the puppet class role::ci::slave::labs applied which eventually call profile::ci::package_builder which in turns reuse the package_builder class used by production. This way CI is more or less aligned with what production uses.

hashar added a comment.EditedMar 19 2019, 8:29 PM

I have cherry picked https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/495681/3 on the CI puppetmaster and I now have:

$ apt-cache policy jenkins-debian-glue
jenkins-debian-glue:
  Installed: 0.20.0~wmf8u1
  Candidate: 0.20.0~wmf8u1
  Version table:
 *** 0.20.0~wmf8u1 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/component/ci amd64 Packages
        100 /var/lib/dpkg/status
     0.18.4-wmf3 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages

Then apt-get update complains with:

W: Duplicate sources.list entry http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/component/ci amd64 Packages (/var/lib/apt/lists/apt.wikimedia.org_wikimedia_dists_jessie-wikimedia_component_ci_binary-amd64_Packages)

Comes from the two files:

/var/lib/apt/lists/apt.wikimedia.org_wikimedia_dists_jessie-wikimedia_InRelease
/var/lib/apt/lists/apt.wikimedia.org_wikimedia_dists_jessie-wikimedia_component_ci_binary-amd64_Packages

I have thus removed the file /etc/apt/sources.list.d/jessie-ci-php55.list and that got rid of the issue :)

Pending the puppet patch to be merged. Otherwise it is essentially solved :]

Change 495681 merged by Jbond:
[operations/puppet@production] Add component/ci wikimedia repository to CI hosts

https://gerrit.wikimedia.org/r/495681

patch is merged, let me know if there is anything elses from my side

hashar closed this task as Resolved.Fri, Mar 22, 11:18 PM

Seems good so far. Thank you very much. I will later look at migrating to Stretch now that I have the package available in apt!