Page MenuHomePhabricator

Upgrade Jenkins from 1.x to latest 2.x
Closed, ResolvedPublic

Description

1.x is no more supported and Jenkins 2.x is a major release. It is not as scary as it sounds though. The reason for the major version bump is a revamped UI and the merge of the Pipeline plugin (a DSL to define jobs).

Will first want to Upgrade Jenkins to 1.651.3 T144105

Changelog: https://jenkins.io/changelog-stable/

Upgrade process: https://wikitech.wikimedia.org/wiki/Jenkins

Upgrade guide: https://jenkins.io/doc/upgrade-guide/

Details

Related Gerrit Patches:
operations/puppet : productionnodepool: pin python-jenkins to 0.4.12+
operations/debs/nodepool : debian0.1.1-wmf6: force jenkins queries to use POST
operations/debs/nodepool : patch-queue/debianWMF: force Jenkins queries to use POST
operations/puppet : productionnodepool: bump python-jenkins

Event Timeline

hashar created this task.Aug 27 2016, 7:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 27 2016, 7:53 PM

I have tested jenkins 1.x and 2.x and there is deffitly nothing stoping us from going to 2.x. One time I accidentally downgraded to 1.x due to me doing apt-get update and apt-get upgrade and it installed the Wikimedia version of jenkins and it worked after without nothing breaking.

+1

hashar triaged this task as High priority.Sep 26 2016, 11:05 AM
hashar moved this task from Untriaged to Backlog on the Continuous-Integration-Infrastructure board.
hashar renamed this task from Upgrade Jenkins from 1.x to 2.7.2 to Upgrade Jenkins from 1.x to latest 2.x.Nov 15 2016, 4:15 PM
Paladox added a comment.EditedNov 15 2016, 4:39 PM

Jenkins 1.x can be updated to 2.x at any time. It can be flawlessly downgraded from 2.x to 1.x and then 1.x to 2.x. Just note that you will need to update all the plugins too.

jenkins 2.31 now includes mobile display. Should be mobile friendly now.

It would be nice to have a way to test the upgrade before it happens and make any tweaks to puppet along the way.

This would also be nice for future upgrades, plugin changes, etc. Like a beta cluster for CI.

Created T150772: Request for CI staging project to explore this idea.

T150771 is to get a spare Jenkins in production. Can later be used to prepare/validate Jenkins upgrades in production environment.

greg added a subscriber: greg.Nov 24 2016, 12:23 AM

I'd like to make the ci-staging environment an explicit blocker for this, OK? (we can talk about it on Monday in our team meeting as well, but, just FYI I want to :) ).

Yes that is implied. @thcipriani and I discussed about it last week and agreed the upgrade requires proper testing. Thus we need a sandbox area to exercise the upgrade which can be done on either of:

  • populate the staging area in labs (made possible since T150772)
  • use a spare production box T150771

Which ever comes first will have the honor to receive a copy of our Jenkins and get used for the Jenkins 2 upgrade test :]

Upstream are updating ssh in https://github.com/jenkinsci/jenkins/pull/2662 so maybe that will include the new ssh standards?

@hashar hi, has there been any progress on this?

hashar updated the task description. (Show Details)Feb 7 2017, 10:23 AM

Been reviewing the upgrade guide and I guess I will just go with the first 2.X LTS : 2.7.4 https://pkg.jenkins.io/debian-stable/binary/jenkins_2.7.4_all.deb

@hashar unless you have some special way that allows you to use <p> on tests config pages then it will not work in jenkins 2.x as far as i know.

@hashar i think upstream are discontinuing support for java 7 and instead going to support java 8. Or that's what it looks like by https://github.com/jenkinsci/jenkins/pull/2724 and https://issues.jenkins-ci.org/browse/JENKINS-36872

Mentioned in SAL (#wikimedia-operations) [2017-05-02T08:04:07Z] <hashar> Upgrading Jenkins to 2.7.4 - T144106

Mentioned in SAL (#wikimedia-operations) [2017-05-02T08:26:07Z] <hashar> Upgrading Jenkins to 2.19.4 - T144106

Mentioned in SAL (#wikimedia-operations) [2017-05-02T08:33:28Z] <hashar> Upgrading Jenkins to 2.32.3 - T144106

Mentioned in SAL (#wikimedia-operations) [2017-05-02T08:40:29Z] <hashar> Upgrading Jenkins to 2.46.2 - T144106

hashar closed this task as Resolved.May 2 2017, 8:59 AM
hashar claimed this task.

We are now running Jenkins LTS 2.46.2

I have also upgraded all plugins and installed the Pipeline set of plugins.

There is some cleanup tasks to add such as https://gerrit.wikimedia.org/r/#/c/351261/

hashar reopened this task as Open.May 2 2017, 9:42 AM

I had to rollback to 2.32.3 , Nodepool was no more able to add instances as Jenkins slaves.

hashar added a comment.May 2 2017, 9:55 AM

That is due to 2.46.2 to prevent cross script forgery ( https://jenkins.io/security/advisory/2017-04-26/ ). Looks like Nodepool old python-jenkins does not support it.

Mentioned in SAL (#wikimedia-operations) [2017-05-02T10:12:01Z] <hashar> Upgrading Jenkins to 2.46.1 - T144106

Change 351280 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] nodepool: bump python-jenkins

https://gerrit.wikimedia.org/r/351280

Change 351280 merged by Muehlenhoff:
[operations/puppet@production] nodepool: bump python-jenkins

https://gerrit.wikimedia.org/r/351280

So that progressed nicely, thanks to Moritz for the support.

We are now at Jenkins 2.46.1.

Further upgrade to 2.46.2 is blocked because Jenkins requires some requests to be POST ed and with a token. That prevents Nodepool from reaching Jenkins.

The python-jenkins version we have in jessie is too old and the one from jessie-backports is not recent enough: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861656

So this is on hold till we get python-jenkins upgraded.

Paladox added a comment.EditedMay 2 2017, 12:09 PM

Should we file a bug upstream at Jenkins or on openstack?

Is it the python-Jenkins upgrade that's blocking this or is it both that and the post thingy?

@hashar you probably want to adjust your bug report to say 0.4.14 instead of 0.4.12. This https://github.com/openstack/python-jenkins/commit/0d7b59f5d24ece9d697ddb47d2785c432b94e9c3 is the fix your most likely looking to fix when our URL has /ci/ in it.

I've built a custom package of python-jenkins 0.4.14 for jessie-wikimedia, I'll deploy it tomorrow with Antoine.

Any future upgrades to any more jenkins its releases will be blocked by T162828 (i mean a new upcoming lts release in the summer)

Mentioned in SAL (#wikimedia-operations) [2017-05-03T07:53:26Z] <hashar> Upgrading Jenkins 2.46.1 -> 2.46.2 - T144106

Mentioned in SAL (#wikimedia-operations) [2017-05-03T08:01:23Z] <hashar> Rolling back Jenkins 2.46.2 -> 2.46.1 - T144106

Change 351606 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/nodepool@patch-queue/debian] WMF: force Jenkins queries to use POST

https://gerrit.wikimedia.org/r/351606

Change 351606 merged by Hashar:
[operations/debs/nodepool@patch-queue/debian] WMF: force Jenkins queries to use POST

https://gerrit.wikimedia.org/r/351606

Change 351608 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/nodepool@debian] 0.1.1-wmf6: force jenkins queries to use POST

https://gerrit.wikimedia.org/r/351608

Change 351608 merged by Muehlenhoff:
[operations/debs/nodepool@debian] 0.1.1-wmf6: force jenkins queries to use POST

https://gerrit.wikimedia.org/r/351608

Mentioned in SAL (#wikimedia-operations) [2017-05-03T11:23:02Z] <hashar> Upgrading Jenkins 2.46.1 -> 2.46.2 - T144106

hashar closed this task as Resolved.May 3 2017, 11:29 AM

We are now running Jenkins latest LTS version 2.46.2.

An issue was Nodepool not being to add/remove instances in Jenkins. I thought it was due to an obsolete python-jenkins package which @MoritzMuehlenhoff kindly rebuild to use the latest version. Turns out Nodepool overrides the python-jenkins methods to deal with nodes and I had to hack that part to force POST requests.

Nodepool package got rebuild with the patch and is now at 0.1.1-wmf7.

Change 366217 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] nodepool: pin python-jenkins to 0.4.12

https://gerrit.wikimedia.org/r/366217

Change 366217 merged by Muehlenhoff:
[operations/puppet@production] nodepool: pin python-jenkins to 0.4.12

https://gerrit.wikimedia.org/r/366217