Change 484975 had a related patch set uploaded (by Hashar; owner: Legoktm):
[integration/config@master] php-composer-security-docker: Use composer:0.1.8 image

https://gerrit.wikimedia.org/r/484975

My patches are in v0.20.0. I used to have the package build from operations/debs/jenkins-debian-glue which had our fork but it is no more needed.

Instead we can just rebuild upstream version which has landed in Debian testing/unstable https://packages.debian.org/buster/jenkins-debian-glue

Looks like we need:

http://deb.debian.org/debian/pool/main/j/jenkins-debian-glue/jenkins-debian-glue_0.20.0.dsc
http://deb.debian.org/debian/pool/main/j/jenkins-debian-glue/jenkins-debian-glue_0.20.0.tar.xz

Or maybe reprepro can let us import it directly. The binary packages should be added to jessie-wikimedia/main and if at all possible to stretch-wikimedia/main as well.

I have managed to build the package for both jessie and stretch without any issues! :)

To clarify from discussions I had:

• the packages are not installed on production servers, solely on a few WMCS instances used for CI.
• they are only used for CI needs
• the source package is a Debian native package, seems to qualify for the main component.

Sounds like SRE can now get the source package from Debian.org, rebuild it for Jessie and Stretch and upload the resulting binaries to jessie-wikimedia/main and stretch-wikimedia/main :]

In T212774#4897016, @hashar wrote:

• the source package is a Debian native package, seems to qualify for the main component.

We have component/ci for a while, shouldn't it be uploaded there?

On Jessie we had it in jessie-wikimedia/main, which predates the introduction of component/xxx in our apt repositories. So we have:

$ apt-cache policy jenkins-debian-glue
jenkins-debian-glue:
  Installed: 0.18.4-wmf3
  Candidate: 0.18.4-wmf3
  Version table:
 *** 0.18.4-wmf3 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages
        100 /var/lib/dpkg/status

Eventually upstream got the package uploaded and it has made it in Stretch. I guess under stretch/main. So yeah we should align with the new way of doing things and add it to component/ci for both distributions.

The only question I have is: on Stretch would apt default to the latest version or will I need an apt preference to select the version from component/ci? Or in other worlds, are packages from component/xxx taking over or are their optional? ;)

Change 495681 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] Add component/ci wikimedia repository to CI hosts

https://gerrit.wikimedia.org/r/495681

i have built this and added it to {jessie,stretch}-wikimedia in component/ci. It appears that the CI servers did not have components/ci configured so i have also created a change to add the repo to those servers. As far as i can tell this only needs to go to contint2001.wikimedia.org and contint1001.wikimedia.org, which both run Jessie. Have i missed some nodes which may also need this repo?

Thanks John

@hashar i have taken another look at the patch i created yesterday and i now think it is incorrect. As far as i can tell profile::ci::package_builder which is the only class that attempts to install jenkins-debian-glue is never run on any server. profile::ci::package_builder is included in role::ci::slave::labs however

it/puppet $ grep -lr profile::ci::package_builder ./               
./modules/role/manifests/ci/slave/labs.pp
./modules/profile/manifests/ci/package_builder.pp
git/puppet $
git/puppet $ grep -lr 'role::ci::slave::labs[^:]' ./                    
./modules/role/manifests/ci/slave/labs.pp
git/puppet $
git/puppet $ grep ci::slave::labs manifests/site.pp
git/puppet $

Further to this cumin produces no results

jbond@cumin1001:~$ sudo cumin 'C:profile::ci::package_builder'
No hosts found that matches the query
jbond@cumin1001:~$ sudo cumin 'C:ci::slave::labs'
No hosts found that matches the query
jbond@cumin1001:~$ sudo cumin 'R:Package = jenkins-debian-glue'
No hosts found that matches the query

As such I'm unsure how best to proceed. Which servers need the jenkins-debian-glue package?

Thanks John

> As far as i can tell this only needs to go to contint2001.wikimedia.org and contint1001.wikimedia.org

Those run the CI stack (Jenkins, Zuul etc) but they do not build the packages. Jenkins connects to WMCS instances that then run the actual jobs. Those instances are in the integration WMCS tenant and they are provisioned with puppet classes which are applied in Horizon.

The instances building the debian packages are:

They have the puppet class role::ci::slave::labs applied which eventually call profile::ci::package_builder which in turns reuse the package_builder class used by production. This way CI is more or less aligned with what production uses.

I have cherry picked https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/495681/3 on the CI puppetmaster and I now have:

$ apt-cache policy jenkins-debian-glue
jenkins-debian-glue:
  Installed: 0.20.0~wmf8u1
  Candidate: 0.20.0~wmf8u1
  Version table:
 *** 0.20.0~wmf8u1 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/component/ci amd64 Packages
        100 /var/lib/dpkg/status
     0.18.4-wmf3 0
       1001 http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/main amd64 Packages

Then apt-get update complains with:

W: Duplicate sources.list entry http://apt.wikimedia.org/wikimedia/ jessie-wikimedia/component/ci amd64 Packages (/var/lib/apt/lists/apt.wikimedia.org_wikimedia_dists_jessie-wikimedia_component_ci_binary-amd64_Packages)

Comes from the two files:

/var/lib/apt/lists/apt.wikimedia.org_wikimedia_dists_jessie-wikimedia_InRelease
/var/lib/apt/lists/apt.wikimedia.org_wikimedia_dists_jessie-wikimedia_component_ci_binary-amd64_Packages

I have thus removed the file /etc/apt/sources.list.d/jessie-ci-php55.list and that got rid of the issue :)

Pending the puppet patch to be merged. Otherwise it is essentially solved :]

Change 495681 merged by Jbond:
[operations/puppet@production] Add component/ci wikimedia repository to CI hosts

https://gerrit.wikimedia.org/r/495681

patch is merged, let me know if there is anything elses from my side

In T212774#5049731, @hashar wrote:

Seems good so far. Thank you very much. I will later look at migrating to Stretch now that I have the package available in apt!

Filled as T224943