Page MenuHomePhabricator

Align jenkins-job-builder with upstream
Closed, ResolvedPublic

Description

We use a fork of jenkins-job-builder via the Gerrit repository integration/jenkins-job-builder.git Gerrit repository.

It had a cherry pick of a patch I have made years ago: https://review.opendev.org/#/c/471030/1 which I have dropped some minutes ago (it is no more needed). Our fork is now pointing at tag 2.0.2.

integration/config.git uses:

jenkins-jobs-requirements.txt
git+https://gerrit.wikimedia.org/r/integration/jenkins-job-builder.git@master#egg=jenkins_job_builder

We should just use pypi for now until we gotta fork again. Then proceed with catching up upstream.

Details

Related Gerrit Patches:
integration/config : masterMake tox notice dependency changes
integration/config : masterUpgrade jjb to 3.0.2
integration/config : masterUpgrade jjb to 2.10.1
integration/config : masterUpgrade jjb to 2.9.1
integration/config : masterUpgrade jjb to 2.4.0
integration/config : masterUpgrade jjb to 2.3.0
integration/config : masterUpgrade jjb to 2.2.1
integration/config : masterUpgrade jjb to 2.0.10
integration/config : masterUpgrade jjb to 2.0.9
integration/config : masterUpgrade jjb to 2.9.1
integration/config : masterConstrain PyYAML<4
integration/config : masterUse jenkins-job-builder from pypi

Event Timeline

hashar created this task.Oct 21 2019, 9:54 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 21 2019, 9:54 AM

Change 544851 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Use jenkins-job-builder from pypi

https://gerrit.wikimedia.org/r/544851

hashar added a subtask: Restricted Task.Oct 21 2019, 10:25 AM

jenkins-job-builder 2.0.3 fails due to our outdated usage of postbuildscript. We would need the plugin to be upgraded first: T188398.

hashar claimed this task.Oct 21 2019, 1:38 PM
hashar triaged this task as Normal priority.

Change 544851 merged by jenkins-bot:
[integration/config@master] Use jenkins-job-builder from pypi

https://gerrit.wikimedia.org/r/544851

We now use 2.0.2 from pypi.

2.0.3 upgrade is blocked on upgrading the PostBuildScript plugin.

hashar closed subtask Restricted Task as Resolved.Thu, Oct 24, 10:22 AM

integration/config.git jenkins-jobs-requirements.txt now uses 2.0.3

I have updated integration/jenkins-job-builder.git to 2.0.3 as well.

2.0.10 introduces a depth parameter for git shallow clone and it always explicitly set with a default of 1. So the JJB diff shows:

Comparing...
diff -u output-prev/mediawiki-core-doxygen-docker output/mediawiki-core-doxygen-docker
--- output-prev/mediawiki-core-doxygen-docker	2019-10-24 15:31:32.696409153 +0200
+++ output/mediawiki-core-doxygen-docker	2019-10-24 15:33:50.426023111 +0200
@@ -56,6 +56,7 @@
       </hudson.plugins.git.extensions.impl.LocalBranch>
       <hudson.plugins.git.extensions.impl.CloneOption>
         <shallow>false</shallow>
+        <depth>1</depth>
         <noTags>true</noTags>
       </hudson.plugins.git.extensions.impl.CloneOption>
       <hudson.plugins.git.extensions.impl.WipeWorkspace/>
diff -u output-prev/mediawiki-core-phpmetrics-docker output/mediawiki-core-phpmetrics-docker
--- output-prev/mediawiki-core-phpmetrics-docker	2019-10-24 15:31:32.716411712 +0200
+++ output/mediawiki-core-phpmetrics-docker	2019-10-24 15:33:50.450026181 +0200
@@ -55,6 +55,7 @@
       </hudson.plugins.git.extensions.impl.LocalBranch>
       <hudson.plugins.git.extensions.impl.CloneOption>
         <shallow>true</shallow>
+        <depth>1</depth>
       </hudson.plugins.git.extensions.impl.CloneOption>
       <hudson.plugins.git.extensions.impl.WipeWorkspace/>
     </extensions>
diff -u output-prev/operations-puppet-wmf-style-guide output/operations-puppet-wmf-style-guide
--- output-prev/operations-puppet-wmf-style-guide	2019-10-24 15:31:33.152467477 +0200
+++ output/operations-puppet-wmf-style-guide	2019-10-24 15:33:50.814072739 +0200
@@ -46,6 +46,7 @@
       </hudson.plugins.git.extensions.impl.RelativeTargetDirectory>
       <hudson.plugins.git.extensions.impl.CloneOption>
         <shallow>false</shallow>
+        <depth>1</depth>
         <noTags>true</noTags>
       </hudson.plugins.git.extensions.impl.CloneOption>
       <hudson.plugins.git.extensions.impl.SubmoduleOption>

There is a warning about --output writing jobs differently. So that instead of writing a file foobar, it does foobar/config.xml. So some script and doc will need to be adjusted. The warning shows up as:

Warning:jenkins_jobs.builder:(Deprecated)
The default output behavior of jenkins-jobs test when given the --output flag will change in JJB 3.0.
Instead of writing jobs to OUTPUT/jobname; they will be written to OUTPUT/jobname/config.xml.
The new behavior can be enabled by the passing --config-xml parameter.

2.3.0 has AnsiColor to always set the colormap to xterm.

4cce3d6a78fbe4e1de39299f77acf07943c61a3b

2.4.0 adds some extra XML parameters for the copyartifact property. That only affects beta-publish-deb:

diff -u output-prev/beta-publish-deb output/beta-publish-deb
--- output-prev/beta-publish-deb	2019-10-24 15:50:08.455101260 +0200
+++ output/beta-publish-deb	2019-10-24 15:53:53.315858163 +0200
@@ -20,6 +20,8 @@
       <optional>false</optional>
       <doNotFingerprintArtifacts>false</doNotFingerprintArtifacts>
       <parameters/>
+      <exclude/>
+      <resultVariableSuffix/>
       <selector class="hudson.plugins.copyartifact.TriggeredBuildSelector">
         <fallbackToLastSuccessful>false</fallbackToLastSuccessful>
       </selector>

2.9.0 makes the --output related warning to only show once ( e07ad13ad0173a4f5e6ab899cfb6d78861c26d42 ).

Change 545861 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 2.9.1

https://gerrit.wikimedia.org/r/545861

Change 545868 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Contrain PyYAML<4

https://gerrit.wikimedia.org/r/545868

Almost ready to reach latest version available :-]

Change 545868 merged by jenkins-bot:
[integration/config@master] Constrain PyYAML<4

https://gerrit.wikimedia.org/r/545868

Change 546113 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 2.0.9

https://gerrit.wikimedia.org/r/546113

Change 546114 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 2.0.10

https://gerrit.wikimedia.org/r/546114

Change 546115 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 2.2.1

https://gerrit.wikimedia.org/r/546115

Change 545861 abandoned by Hashar:
Upgrade jjb to 2.9.1

Reason:
I am splitting this change.

https://gerrit.wikimedia.org/r/545861

Change 546121 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 2.3.0

https://gerrit.wikimedia.org/r/546121

Change 546130 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 2.9.1

https://gerrit.wikimedia.org/r/546130

Change 546113 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 2.0.9

https://gerrit.wikimedia.org/r/546113

Change 546114 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 2.0.10

https://gerrit.wikimedia.org/r/546114

Change 546115 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 2.2.1

https://gerrit.wikimedia.org/r/546115

Change 546121 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 2.3.0

https://gerrit.wikimedia.org/r/546121

Mentioned in SAL (#wikimedia-releng) [2019-10-25T09:19:29Z] <hashar> Reconfiguring all Jenkins jobs defined by JJB due to upgrade to jjb 2.9.1 # T236036

Mentioned in SAL (#wikimedia-releng) [2019-10-25T09:20:12Z] <hashar> Updated integration/jenkins-job-builder.git to 2.9.1 # T236036

Change 546128 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 2.4.0

https://gerrit.wikimedia.org/r/546128

Change 546130 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 2.9.1

https://gerrit.wikimedia.org/r/546130

hashar closed this task as Resolved.Fri, Oct 25, 9:21 AM

Hurrah

hashar reopened this task as Open.Mon, Oct 28, 4:50 PM

I have had an outdated fork on my laptop that was still pointing to an obsolete upstream git. They have since moved to opendev.org which has the tags:

2.9.1   # the one in integration/config
2.10.0
2.10.1
3.0.0
3.0.1
3.0.2

One breaking change is --config-xml which has been solved by T236685: Migrate JJB usage --config-xml

Change 546653 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 2.10.1

https://gerrit.wikimedia.org/r/546653

Change 546653 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 2.10.1

https://gerrit.wikimedia.org/r/546653

Change 546663 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 3.0.2

https://gerrit.wikimedia.org/r/546663

Change 546663 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 3.0.2

https://gerrit.wikimedia.org/r/546663

Change 546698 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Have notice dependency changes

https://gerrit.wikimedia.org/r/546698

Last thing is that when changing the JJB version in jenkins-job-requirements.txt, tox does not notice the different and would not rebuild the virtual environment. That is tox bug: https://github.com/tox-dev/tox/issues/149

We have to move dependencies and inline them in tox.ini :]

Change 546698 merged by jenkins-bot:
[integration/config@master] Make tox notice dependency changes

https://gerrit.wikimedia.org/r/546698

hashar closed this task as Resolved.Mon, Oct 28, 8:20 PM

Thank you @Jdforrester-WMF for the reviews / support etc!