1.x is no more supported and Jenkins 2.x is a major release. It is not as scary as it sounds though. The reason for the major version bump is a revamped UI and the merge of the Pipeline plugin (a DSL to define jobs).
Will first want to Upgrade Jenkins to 1.651.3 T144105
Changelog: https://jenkins.io/changelog-stable/
Upgrade process: https://wikitech.wikimedia.org/wiki/Jenkins
Upgrade guide: https://jenkins.io/doc/upgrade-guide/
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T100518 Reenable ssh MAC/KEX hardening on beta cluster and integration labs project | |||
| Resolved | hashar | T100509 Jenkins master / client ssh connection fails due to missing ssh algorithm | |||
| Resolved | Paladox | T103351 Jenkins trilead-ssh2 doesn't support our MAC/KEX algorithms | |||
| Resolved | hashar | T155840 Install the BlueOcean plugin on jenkins | |||
| Resolved | hashar | T144106 Upgrade Jenkins from 1.x to latest 2.x | |||
| Declined | None | T144105 Upgrade Jenkins to 1.651.3 | |||
| Resolved | MoritzMuehlenhoff | T157429 Upload Jenkins LTS v2.46.2 to jessie-wikimedia/third-party |
I have tested jenkins 1.x and 2.x and there is deffitly nothing stoping us from going to 2.x. One time I accidentally downgraded to 1.x due to me doing apt-get update and apt-get upgrade and it installed the Wikimedia version of jenkins and it worked after without nothing breaking.
+1
Jenkins 1.x can be updated to 2.x at any time. It can be flawlessly downgraded from 2.x to 1.x and then 1.x to 2.x. Just note that you will need to update all the plugins too.
jenkins 2.31 now includes mobile display. Should be mobile friendly now.
It would be nice to have a way to test the upgrade before it happens and make any tweaks to puppet along the way.
This would also be nice for future upgrades, plugin changes, etc. Like a beta cluster for CI.
Created T150772: Request for CI staging project to explore this idea.
T150771 is to get a spare Jenkins in production. Can later be used to prepare/validate Jenkins upgrades in production environment.
I'd like to make the ci-staging environment an explicit blocker for this, OK? (we can talk about it on Monday in our team meeting as well, but, just FYI I want to :) ).
Yes that is implied. @thcipriani and I discussed about it last week and agreed the upgrade requires proper testing. Thus we need a sandbox area to exercise the upgrade which can be done on either of:
Which ever comes first will have the honor to receive a copy of our Jenkins and get used for the Jenkins 2 upgrade test :]
Upstream are updating ssh in https://github.com/jenkinsci/jenkins/pull/2662 so maybe that will include the new ssh standards?
Been reviewing the upgrade guide and I guess I will just go with the first 2.X LTS : 2.7.4 https://pkg.jenkins.io/debian-stable/binary/jenkins_2.7.4_all.deb
@hashar unless you have some special way that allows you to use <p> on tests config pages then it will not work in jenkins 2.x as far as i know.
@hashar i think upstream are discontinuing support for java 7 and instead going to support java 8. Or that's what it looks like by https://github.com/jenkinsci/jenkins/pull/2724 and https://issues.jenkins-ci.org/browse/JENKINS-36872
Mentioned in SAL (#wikimedia-operations) [2017-05-02T08:04:07Z] <hashar> Upgrading Jenkins to 2.7.4 - T144106
Mentioned in SAL (#wikimedia-operations) [2017-05-02T08:26:07Z] <hashar> Upgrading Jenkins to 2.19.4 - T144106
Mentioned in SAL (#wikimedia-operations) [2017-05-02T08:33:28Z] <hashar> Upgrading Jenkins to 2.32.3 - T144106
Mentioned in SAL (#wikimedia-operations) [2017-05-02T08:40:29Z] <hashar> Upgrading Jenkins to 2.46.2 - T144106
We are now running Jenkins LTS 2.46.2
I have also upgraded all plugins and installed the Pipeline set of plugins.
There is some cleanup tasks to add such as https://gerrit.wikimedia.org/r/#/c/351261/
I had to rollback to 2.32.3 , Nodepool was no more able to add instances as Jenkins slaves.
That is due to 2.46.2 to prevent cross script forgery ( https://jenkins.io/security/advisory/2017-04-26/ ). Looks like Nodepool old python-jenkins does not support it.
Mentioned in SAL (#wikimedia-operations) [2017-05-02T10:12:01Z] <hashar> Upgrading Jenkins to 2.46.1 - T144106
Change 351280 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] nodepool: bump python-jenkins
Change 351280 merged by Muehlenhoff:
[operations/puppet@production] nodepool: bump python-jenkins
So that progressed nicely, thanks to Moritz for the support.
We are now at Jenkins 2.46.1.
Further upgrade to 2.46.2 is blocked because Jenkins requires some requests to be POST ed and with a token. That prevents Nodepool from reaching Jenkins.
The python-jenkins version we have in jessie is too old and the one from jessie-backports is not recent enough: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861656
So this is on hold till we get python-jenkins upgraded.
Should we file a bug upstream at Jenkins or on openstack?
Is it the python-Jenkins upgrade that's blocking this or is it both that and the post thingy?
@hashar you probably want to adjust your bug report to say 0.4.14 instead of 0.4.12. This https://github.com/openstack/python-jenkins/commit/0d7b59f5d24ece9d697ddb47d2785c432b94e9c3 is the fix your most likely looking to fix when our URL has /ci/ in it.
I've built a custom package of python-jenkins 0.4.14 for jessie-wikimedia, I'll deploy it tomorrow with Antoine.
Any future upgrades to any more jenkins its releases will be blocked by T162828 (i mean a new upcoming lts release in the summer)
Mentioned in SAL (#wikimedia-operations) [2017-05-03T07:53:26Z] <hashar> Upgrading Jenkins 2.46.1 -> 2.46.2 - T144106
Mentioned in SAL (#wikimedia-operations) [2017-05-03T08:01:23Z] <hashar> Rolling back Jenkins 2.46.2 -> 2.46.1 - T144106
Change 351606 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/nodepool@patch-queue/debian] WMF: force Jenkins queries to use POST
Change 351606 merged by Hashar:
[operations/debs/nodepool@patch-queue/debian] WMF: force Jenkins queries to use POST
Change 351608 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/debs/nodepool@debian] 0.1.1-wmf6: force jenkins queries to use POST
Change 351608 merged by Muehlenhoff:
[operations/debs/nodepool@debian] 0.1.1-wmf6: force jenkins queries to use POST
Mentioned in SAL (#wikimedia-operations) [2017-05-03T11:23:02Z] <hashar> Upgrading Jenkins 2.46.1 -> 2.46.2 - T144106
We are now running Jenkins latest LTS version 2.46.2.
An issue was Nodepool not being to add/remove instances in Jenkins. I thought it was due to an obsolete python-jenkins package which @MoritzMuehlenhoff kindly rebuild to use the latest version. Turns out Nodepool overrides the python-jenkins methods to deal with nodes and I had to hack that part to force POST requests.
Nodepool package got rebuild with the patch and is now at 0.1.1-wmf7.
Change 366217 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/puppet@production] nodepool: pin python-jenkins to 0.4.12
Change 366217 merged by Muehlenhoff:
[operations/puppet@production] nodepool: pin python-jenkins to 0.4.12