Page MenuHomePhabricator

Review process to fetch Jenkins Debian package from upstream
Closed, ResolvedPublic

Description

The Jenkins project builds Debian package which we import with reprepro. The documentation listed at https://wikitech.wikimedia.org/wiki/Jenkins#Get_the_package

The documentation there seems outdated since it still refers to jessie but we use stretch/buster. There is no mention of the component/ci component.

The Alternative upgrade method section bypasses reprepro. Apparently as a workaround to avoid updating unrelated packages.

I would like to revisit the process to import the Jenkins LTS package with a single way of clear instructions to import them (a runbook?). That would in turn lets anyone from SRE able to handle the operation, typically via the clinic duty.

Event Timeline

hashar created this task.Aug 12 2020, 8:19 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 12 2020, 8:19 PM
hashar triaged this task as Medium priority.Aug 12 2020, 8:19 PM

No hurry. I filed it cause we ended up importing the non LTS version today and having two methods looks odd.

Dzahn added a comment.Aug 17 2020, 6:36 PM

the Alternative upgrade method section bypasses reprepro

This is not accurate. It uses "sudo -i reprepro ...".

Dzahn added a comment.Aug 17 2020, 6:39 PM

No hurry. I filed it cause we ended up importing the non LTS version today and having two methods looks odd.

Turns out it was actually good that we are on a newer version than LTS.

I don't see yet which actual problem would be solved by converting reprepro commands to a runbook.

hashar claimed this task.Sep 25 2020, 8:56 AM

TLDR:

  • Remove reference to Jessie/Stretch ini the config file and on wiki
  • reprepro -v -C thirdparty/ci checkupdate buster-wikimedia
  • reprepro -v -C thirdparty/ci update buster-wikimedia

I finally went to instal reprepro on my machine with some dummy configuration files:

conf/distributions
Origin: Wikimedia
Label: Wikimedia
Suite: buster-wikimedia
Codename: buster-wikimedia
AlsoAcceptFor: buster buster-backports
Version: 10
Architectures: source amd64 i386
Components: main
# sorted list, leading whitespace matters!
 thirdparty/ci
UDebComponents: main
Update:
# sorted list, leading whitespace matters!
 jenkins
Description: Wikimedia packages for Debian buster
#SignWith: 9D392D3FFADF18FB
#Log:
# log
updates
# "reprepro update" doesn't work for Jenkins due to https://issues.jenkins-ci.org/browse/INFRA-165
Name: jenkins
Method: http://pkg.jenkins-ci.org/debian-stable/
Suite: binary
Flat: thirdparty/ci
GetInRelease: no
Architectures: all>amd64
VerifyRelease: FCEF32E745F2C3D5
ListShellHook: grep-dctrl -X -S jenkins || [ $? -eq 1 ]

The update configuration for jenkins uses Flat: thirdparty while the one for stretch used Flat: thirdparty/ci. I have changed it in the above config.

We can drop our configuration for Jessie and Stretch since both releases and contint hosts are now running Buster.

I can then check for updates just for that component:

reprepro -v -C thirdparty/ci checkupdate buster-wikimedia
aptmethod got 'http://pkg.jenkins-ci.org/debian-stable/binary/Release'
aptmethod got 'http://pkg.jenkins-ci.org/debian-stable/binary/Release.gpg'
aptmethod got 'http://pkg.jenkins-ci.org/debian-stable/binary/Packages.lzma'
Shutting down aptmethods...
Calculating packages to get...
Updates needed for 'buster-wikimedia|thirdparty/ci|amd64':
'jenkins': newly installed as '2.249.1' (from 'jenkins'):
 files needed: pool/thirdparty/ci/j/jenkins/jenkins_2.249.1_all.deb

Update solely the packages from that component with:

reprepro -v -C thirdparty/ci update buster-wikimedia
aptmethod got 'http://pkg.jenkins-ci.org/debian-stable/binary/Release'
aptmethod got 'http://pkg.jenkins-ci.org/debian-stable/binary/Release.gpg'
aptmethod got 'http://pkg.jenkins-ci.org/debian-stable/binary/Packages.lzma'
Calculating packages to get...
Getting packages...
aptmethod redirects 'http://pkg.jenkins-ci.org/debian-stable/binary/jenkins_2.249.1_all.deb' to 'http://mirrors.jenkins.io/debian-stable/jenkins_2.249.1_all.deb'
aptmethod redirects 'http://mirrors.jenkins.io/debian-stable/jenkins_2.249.1_all.deb' to 'http://ftp-chi.osuosl.org/pub/jenkins/debian-stable/jenkins_2.249.1_all.deb'
aptmethod got 'http://ftp-chi.osuosl.org/pub/jenkins/debian-stable/jenkins_2.249.1_all.deb'
Shutting down aptmethods...
Installing (and possibly deleting) packages...
Exporting indices...

And check again:

reprepro -v -C thirdparty/ci checkupdate buster-wikimedia
aptmethod got 'http://pkg.jenkins-ci.org/debian-stable/binary/Release'
aptmethod got 'http://pkg.jenkins-ci.org/debian-stable/binary/Release.gpg'
Nothing to do found. (Use --noskipold to force processing)

This doesn't solve the issue that the LTS releases cannot be differentiated on the apt package level. If I add the apt source, apt-cache show provides me the ones below (and many, many more dating back to 1.409.1)

So in the case of two overlapping LTS series (and we using the old one), this would still import the more recent one. It's probably an acceptable tradeoff?

Package: jenkins
Architecture: all
Version: 2.249.1
Priority: extra
Section: devel
Maintainer: Kohsuke Kawaguchi <kk@kohsuke.org>
Installed-Size: 65855
Depends: daemon, adduser, procps, psmisc, net-tools
Conflicts: hudson
Replaces: hudson
Filename: binary/jenkins_2.249.1_all.deb
Size: 66030672
MD5sum: f21d392f566d24e9dcb4fe430fbc0284
SHA1: 906f66edec192cd91f7d54d3f8a4281201b165d8
SHA256: 2a5db0ebd9654baf207a94c5355f57ec8890329ff479bf2f97d729b2586e6e08
SHA512: 3fa05ff20b46493aac8ebe67f96938e2c88833945dbefa8f89499bc484542d352dfbe5cbd56cc1f910ad238fd589bb14aa8f1b90d024bf264f7b83cf8152eac0
Homepage: http://jenkins.io/
Description: Jenkins is an open source automation server which enables developers around the world to reliably automate  their development lifecycle processes of all kinds, including build, document, test, package, stage, deployment, static analysis and many more.
 .
 Jenkins is being widely used in areas of Continous Integration, Continuous Delivery, DevOps, and other areas. And it is not only about software, the same automation techniques can be applied in other areas like Hardware Engineering, Embedded Systems, BioTech, etc.
 .
 For information see https://jenkins.io
Description-md5: e13cbf68f91c23410d902c5bb8105b86

Package: jenkins
Architecture: all
Version: 2.235.5
Priority: extra
Section: devel
Maintainer: Kohsuke Kawaguchi <kk@kohsuke.org>
Installed-Size: 64960
Depends: daemon, adduser, procps, psmisc, net-tools
Conflicts: hudson
Replaces: hudson
Filename: binary/jenkins_2.235.5_all.deb
Size: 65576304
MD5sum: 884ceb2300d0af85f97c9e67577d8861
SHA1: 6463c1c77de1438e1b01af6789f3a5cb62ec219d
SHA256: 5c5f5117a1d3914acf6204b6bea2a5abe2432e69e42cade133820f5390377073
SHA512: f3afb46635dcfb17a331684029f5edaa5f13cf5cc3b59e0c81b363dabf6b0a1d896de2e64216c3674438389893ffd2ec4bec9ce1702d0101dd08b6048ba43208
Homepage: http://jenkins.io/
Description: Jenkins is an open source automation server which enables developers around the world to reliably automate  their development lifecycle processes of all kinds, including build, document, test, package, stage, deployment, static analysis and many more.
 .
 Jenkins is being widely used in areas of Continous Integration, Continuous Delivery, DevOps, and other areas. And it is not only about software, the same automation techniques can be applied in other areas like Hardware Engineering, Embedded Systems, BioTech, etc.
 .
 For information see https://jenkins.io
Description-md5: e13cbf68f91c23410d902c5bb8105b86

Change 630095 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove jenkins update targets for jessie/stretch and rename the update config

https://gerrit.wikimedia.org/r/630095

Change 630095 merged by Dzahn:
[operations/puppet@production] Remove jenkins update targets for jessie/stretch and rename the update config

https://gerrit.wikimedia.org/r/630095

This doesn't solve the issue that the LTS releases cannot be differentiated on the apt package level. If I add the apt source, apt-cache show provides me the ones below (and many, many more dating back to 1.409.1)

So in the case of two overlapping LTS series (and we using the old one), this would still import the more recent one. It's probably an acceptable tradeoff?

Package: jenkins
Architecture: all
Version: 2.249.1
... 
Package: jenkins
Architecture: all
Version: 2.235.5
...

So that if we were running 2.235.4 we would end up picking 2.249.1 (the latest) instead of 2.235.5 (the next patch version)? I guess it might be able to differentiate between them with reprepro ListShellHook but I don't get how we would be able to select what kind of upgrade we want. So those would have to be handled manually or maybe by tweaking the updates file.

So that if we were running 2.235.4 we would end up picking 2.249.1 (the latest) instead of 2.235.5 (the next patch version)?

The latest LTS version, not the latest version in general, but yes.

I guess it might be able to differentiate between them with reprepro ListShellHook but I don't get how we would be able to select what kind of upgrade we want. So those would have to be handled manually or maybe by tweaking the updates file.

Yeah, it doesn't seem useful to constantly patch the LTS series in use (given they change ever 2-3 months), we should rather upgrade LTS series early, then we mostly eliminate this risk.

Note we fetch from http://pkg.jenkins-ci.org/debian-stable/ which only has the LTS versions. The weekly ones are in a different distribution: http://pkg.jenkins-ci.org/debian/ . So unless they screw up, we are guaranteed to only fetch LTS ones.

When dealing with security issue, upstream would sometime issue a patch version AND a new LTS. Looking at the changelog ( https://www.jenkins.io/changelog-stable/ ) since January 2017 that happened only three times:

DateLTS Versions
March 20202.204.6 2.222.1
September 20192.176.4 2.190.1
December 20182.138.4 2.150.1

Each of those new LTS had breaking changes they thus went to backport the security fixes to the previous LTS.

I haven't dig in our history of Jenkins upgrades. I guess if we suspect the latest LTS to be potentially damaging, we would manually get the patched one for the current version we use. Since reprepro update does not seem to offer a way to match a specific version, that can be achieved by first deleting it then fetching the version we want with --restrict-binary jenkins=x.yyy.z:

$ reprepro list buster-wikimedia
buster-wikimedia|thirdparty/ci|amd64: jenkins 2.249.1

$ reprepro remove buster-wikimedia jenkins
Exporting indices...
Deleting files no longer referenced...

Then get a specific version:

$ reprepro -V --noskipold --restrict-binary jenkins=2.235.2 update
aptmethod got 'http://ftp-chi.osuosl.org/pub/jenkins/debian-stable/jenkins_2.235.2_all.deb'

$ reprepro list buster-wikimedia
buster-wikimedia|thirdparty/ci|amd64: jenkins 2.235.2

Magic!

I think the exceptional case of double LTS release is rare enough, can you update the wikitech with the steps above and then we can call this resolved from my PoV?

hashar closed this task as Resolved.EditedSep 30 2020, 12:32 PM

Done and published at https://wikitech.wikimedia.org/wiki/Jenkins#Get_the_package

TLDR is:

cd /srv/wikimedia
reprepro checkupdate-C thirdparty/ci --restrict=jenkins  buster-wikimedia
reprepro update -C thirdparty/ci --restrict=jenkins buster-wikimedia

Danke Schon!