Page MenuHomePhabricator

Upgrade Jenkins to 2.277.x
Closed, ResolvedPublic

Description

DebMonitor for Jenkins package

Jenkins is an open source automation server which enables developers around the world to reliably build, test, and deploy their software.
The following releases contain fixes for security vulnerabilities:

  • Jenkins 2.287
  • Jenkins LTS 2.277.2
  • Micro Focus Application Automation Tools Plugin 6.8
  • promoted builds Plugin 3.9.1

Summaries of the vulnerabilities are below. More details, severity, and attribution can be found here:
https://www.jenkins.io/security/advisory/2021-04-07/

We run the previous LTS (2.263.x). The next LTS has a couple of highlighted changes:

  • Configuration page should change from <table> to <div>.
  • Plugins should be upgraded ahead of the upgrade, notably the LDAP one

So it is not entirely trivial

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2021-04-07T13:39:31Z] <moritzm> imported jenkins 2.277.2 to apt.wikimedia.org (thirdparty/ci) T279033

I am on limited hours this week.

We are not affected by the security issues reported in https://www.jenkins.io/security/advisory/2021-04-07/ since:

  • our user rights are very restricted or admins already have all permissions.
  • we don't have the affected plugins

The devil is the 2.277.1 LTS upgrade, notably that some plugins require to be upgraded in advance but I haven't looked them up. Looks like the table to check are:

They might well break some stuff here and there, I am notably worried about the Gearman plugin.

So looks like it is fine for now, but we have to read the 2.277 migration guides, apply the recommendation and after upgrade carefully monitor the setup.

There's another Jenkins security release, which is also dependant on this update:

https://www.jenkins.io/security/advisory/2021-04-20/:

Denial of service vulnerability in bundled Jetty
JENKINS-65280 / CVE-2021-28165

Jenkins bundles Winstone-Jetty, a wrapper around Jetty, to act as HTTP and servlet server when started using java -jar jenkins.war. This is how Jenkins is run when using any of the installers or packages, but not when run using servlet containers such as Tomcat.

Jenkins 2.285 and earlier, LTS 2.277.2 and earlier bundles Jetty 9.4.38 or earlier with multiple security vulnerabilities, including CVE-2021-28165. This vulnerability may allow unauthenticated attackers to cause a denial of service if Winstone-Jetty is configured to handle SSL/TLS connections.

Jenkins LTS 2.277.3 updates the bundled Jetty to 9.4.39. Jetty was already previously updated to 9.4.39 in the 2.286 weekly release.
Severity

JENKINS-65280: High

Affected Versions

Jenkins weekly up to and including 2.285
Jenkins LTS up to and including 2.277.2

Fix

Jenkins weekly should be updated to version 2.286
Jenkins LTS should be updated to version 2.277.3

These versions include fixes to the vulnerabilities described above. All prior versions are considered to be affected by these vulnerabilities unless otherwise indicated.

Neat! Looks like that requires Winstone-Jetty is configured to handle SSL/TLS connections but on our setup we use plain HTTP (and envoy / the cache infra for encryption. So I am guessing we are not affected.

I am currently reviewing the upgrade notes for Spring/Xstream and the <table> to <div> HTML change. I guess this time I will need a backup of the plugins/config and have a written plan to conduct the upgrade :-\

The note about the LDAP plugin is that we must have 1.26 installed which is the latest still compatible with the 2.263.3 Jenkins we are running and has:

Prepared forward compatibility with JEP-227
📦 Dependency updates

  • Compatibility with Spring Security (#48) @jglick

I went ahead and upgraded the release Jenkins and LDAP authentication is still working!

hashar triaged this task as High priority.
hashar updated the task description. (Show Details)

The CI Jenkins upgrade apparently went fine \o/

@MoritzMuehlenhoff I guess we can now import Jenkins 2.277.3 to buster-wikimedia/thirdparty/ci and upgrade again ;]

The CI Jenkins upgrade apparently went fine \o/

@MoritzMuehlenhoff I guess we can now import Jenkins 2.277.3 to buster-wikimedia/thirdparty/ci and upgrade again ;]

Done, just imported the 2.277.3 package :-)

Great. The only issue I have encountered was with the Token Macro plugin on the CI Jenkins which I have somehow forgot to upgrade.

I have upgraded all the Jenkins to 2.277.3! Danke Schon!