Page MenuHomePhabricator

Update jenkins job builder
Closed, ResolvedPublic

Description

Steps to reproduce

  • No steps, just a reminder to update the repo

Update https://phabricator.wikimedia.org/diffusion/CIJJ/ with https://github.com/openstack-infra/jenkins-job-builder please

Event Timeline

The process I follow is:

  • generate the XML config with the current jb
  • git rebase <upstream_commit>
  • generate XML config and do a diff -ur

If all happy, push (maybe force push if we ahve local hacks) to Gerrit.

  • Send a dummy patch to integration/config and review the jjb diff again.
  • Regenerate jobs that are altered (optionally)

Announce to QA list so folks remember to update.

The XML configuration for the build timeout is wrong (T217403). Version 1.14 of the plugin is binary incompatible and JJB was always generating the old XML version. It is slightly broken.

It got fixed by https://review.openstack.org/#/c/387799/ which fix the name and default to the new style configuration introduced by 1.14.

$ git describe --tags --contains ecf3933dc2ce6d0eeca3e73946f62fbec606a36d
2.0.2~3^2

Rebasing our fork to 2.0.2 let us get rid of a hack to require python-jenkins >= 0.4.14:

commit 4bab376c7e62ec4b95c38c03f87ca5179a338e40
Author: Antoine Musso <hashar@free.fr>
Date:   Mon Nov 2 10:14:51 2015 +0100

    [WMF] pin python-jenkins >=0.4.14
    
    A regression has been introduced in python-jenkins since 0.4.9 which
    prevent it from interacting with Jenkins setup having a path in the API
    url such as '/ci/api...'.
    
    Got fixed in python-jenkins by 0d7b59f5d24 released with 0.4.14
    
    Change-Id: I4868dd0b7ac9615e6a07588eba1cfff1590716ed

Upstream 705051658fabb8c0e92cd8dbad87015dfdda7f9f released in 2.0.1 now requires >=0.4.15


Based on 2.0.2 I ran a difference of our configuration. There are a few differences introduced by:

e0eb05d7ef2f53718aeb7b8320ff60b760154684 Add messages and categories ignores to warnings publisher - only update the job operations-puppet-wmf-style-guide

If I revert those three patches, there is no differences introduced. So I am going to rebase on that ;)

Mentioned in SAL (#wikimedia-releng) [2019-03-01T14:28:01Z] <hashar> Upgrading integration/jenkins-job-builder to version 2.0.2 + one custom hack rCIJJ11aa5de481af...rCIJJa06d173e98cc # T143731

I looked at further updating JJB but it fails with 2.0.3. I git bisected it to:

$ git describe --tags 5579395bc3c7d4baea9c3792b977f2a2a484004c
2.0.2-3-g5579395b
commit 5579395bc3c7d4baea9c3792b977f2a2a484004c
Author: Thanh Ha <thanh.ha@linuxfoundation.org>
Date:   Thu Dec 28 16:23:57 2017 -0500

    Add support for PostBuildScript 2.x
    
    Updates the PostBuildScript plugin to support the 2.x version which
    had a major overhaul of the XML output for the plugin.
    
    Change-Id: I9ff3161bc50be236013e32f3ca9ad81b38004dc8
    Signed-off-by: Thanh Ha <thanh.ha@linuxfoundation.org>

Which eventually causes:

Traceback (most recent call last):
  File "/home/hashar/projects/integration/jenkins-job-builder/.tox/venv/bin/jenkins-jobs", line 10, in <module>
    sys.exit(main())
  File "/home/hashar/projects/integration/jenkins-job-builder/jenkins_jobs/cli/entry.py", line 146, in main
    jjb.execute()
  File "/home/hashar/projects/integration/jenkins-job-builder/jenkins_jobs/cli/entry.py", line 140, in execute
    ext.obj.execute(self.options, self.jjb_config)
  File "/home/hashar/projects/integration/jenkins-job-builder/jenkins_jobs/cli/subcommand/test.py", line 55, in execute
    options, jjb_config)
  File "/home/hashar/projects/integration/jenkins-job-builder/jenkins_jobs/cli/subcommand/update.py", line 95, in _generate_xmljobs
    xml_jobs = xml_job_generator.generateXML(job_data_list)
  File "/home/hashar/projects/integration/jenkins-job-builder/jenkins_jobs/xml_config.py", line 77, in generateXML
    xml_jobs.append(self._getXMLForJob(job))
  File "/home/hashar/projects/integration/jenkins-job-builder/jenkins_jobs/xml_config.py", line 88, in _getXMLForJob
    self._gen_xml(xml, data)
  File "/home/hashar/projects/integration/jenkins-job-builder/jenkins_jobs/xml_config.py", line 98, in _gen_xml
    module.gen_xml(xml, data)
  File "/home/hashar/projects/integration/jenkins-job-builder/jenkins_jobs/modules/publishers.py", line 7012, in gen_xml
    self.registry.dispatch('publisher', publishers, action)
  File "/home/hashar/projects/integration/jenkins-job-builder/jenkins_jobs/registry.py", line 254, in dispatch
    func(self, xml_parent, component_data)
  File "/home/hashar/projects/integration/jenkins-job-builder/jenkins_jobs/modules/publishers.py", line 3759, in postbuildscript
    for builder in bs_data.get('build-steps'):
TypeError: 'NoneType' object is not iterable
hashar claimed this task.

That last issue got solved by upgrading the plugin and bumping jjb T188398
f2d2d06dc0f4a6c48c1fd49c516f6a7c981143ec

We are now using JJB 3.1.0 which was T236746