Page MenuHomePhabricator
Create Task
Maniphest T236036

Align jenkins-job-builder with upstream
Closed, ResolvedPublic
Actions

Description

We use a fork of jenkins-job-builder via the Gerrit repository integration/jenkins-job-builder.git Gerrit repository.

It had a cherry pick of a patch I have made years ago: https://review.opendev.org/#/c/471030/1 which I have dropped some minutes ago (it is no more needed). Our fork is now pointing at tag 2.0.2.

integration/config.git uses:

jenkins-jobs-requirements.txt
git+https://gerrit.wikimedia.org/r/integration/jenkins-job-builder.git@master#egg=jenkins_job_builder

We should just use pypi for now until we gotta fork again. Then proceed with catching up upstream.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Oct 21 2019, 9:54 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 21 2019, 9:54 AM
hashar added a project: Technical-Debt.Oct 21 2019, 9:55 AM
gerritbot added a comment.Oct 21 2019, 10:06 AM

Change 544851 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Use jenkins-job-builder from pypi

https://gerrit.wikimedia.org/r/544851

gerritbot added a project: Patch-For-Review.Oct 21 2019, 10:06 AM
hashar added a subtask: Restricted Task.Oct 21 2019, 10:25 AM

jenkins-job-builder 2.0.3 fails due to our outdated usage of postbuildscript. We would need the plugin to be upgraded first: T188398.

Jdforrester-WMF subscribed.Oct 21 2019, 10:26 AM
hashar claimed this task.Oct 21 2019, 1:38 PM
hashar triaged this task as Medium priority.
gerritbot added a comment.Oct 21 2019, 2:02 PM

Change 544851 merged by jenkins-bot:
[integration/config@master] Use jenkins-job-builder from pypi

https://gerrit.wikimedia.org/r/544851

Maintenance_bot removed a project: Patch-For-Review.Oct 21 2019, 2:10 PM
hashar added a comment.Oct 21 2019, 2:16 PM

We now use 2.0.2 from pypi.

2.0.3 upgrade is blocked on upgrading the PostBuildScript plugin.

hashar closed subtask T236046: integration-config-jjb-diff-docker should diff using the proper jjb version as Resolved.Oct 21 2019, 2:16 PM
hashar closed subtask Restricted Task as Resolved.Oct 24 2019, 10:22 AM

integration/config.git jenkins-jobs-requirements.txt now uses 2.0.3

I have updated integration/jenkins-job-builder.git to 2.0.3 as well.

Jdforrester-WMF edited projects, added Release-Engineering-Team-TODO (201910); removed Release-Engineering-Team-TODO.Oct 24 2019, 1:37 PM

Resolved?

hashar added a comment.Oct 24 2019, 1:43 PM

2.0.10 introduces a depth parameter for git shallow clone and it always explicitly set with a default of 1. So the JJB diff shows:

Comparing...
diff -u output-prev/mediawiki-core-doxygen-docker output/mediawiki-core-doxygen-docker
--- output-prev/mediawiki-core-doxygen-docker	2019-10-24 15:31:32.696409153 +0200
+++ output/mediawiki-core-doxygen-docker	2019-10-24 15:33:50.426023111 +0200
@@ -56,6 +56,7 @@
       </hudson.plugins.git.extensions.impl.LocalBranch>
       <hudson.plugins.git.extensions.impl.CloneOption>
         <shallow>false</shallow>
+        <depth>1</depth>
         <noTags>true</noTags>
       </hudson.plugins.git.extensions.impl.CloneOption>
       <hudson.plugins.git.extensions.impl.WipeWorkspace/>
diff -u output-prev/mediawiki-core-phpmetrics-docker output/mediawiki-core-phpmetrics-docker
--- output-prev/mediawiki-core-phpmetrics-docker	2019-10-24 15:31:32.716411712 +0200
+++ output/mediawiki-core-phpmetrics-docker	2019-10-24 15:33:50.450026181 +0200
@@ -55,6 +55,7 @@
       </hudson.plugins.git.extensions.impl.LocalBranch>
       <hudson.plugins.git.extensions.impl.CloneOption>
         <shallow>true</shallow>
+        <depth>1</depth>
       </hudson.plugins.git.extensions.impl.CloneOption>
       <hudson.plugins.git.extensions.impl.WipeWorkspace/>
     </extensions>
diff -u output-prev/operations-puppet-wmf-style-guide output/operations-puppet-wmf-style-guide
--- output-prev/operations-puppet-wmf-style-guide	2019-10-24 15:31:33.152467477 +0200
+++ output/operations-puppet-wmf-style-guide	2019-10-24 15:33:50.814072739 +0200
@@ -46,6 +46,7 @@
       </hudson.plugins.git.extensions.impl.RelativeTargetDirectory>
       <hudson.plugins.git.extensions.impl.CloneOption>
         <shallow>false</shallow>
+        <depth>1</depth>
         <noTags>true</noTags>
       </hudson.plugins.git.extensions.impl.CloneOption>
       <hudson.plugins.git.extensions.impl.SubmoduleOption>

There is a warning about --output writing jobs differently. So that instead of writing a file foobar, it does foobar/config.xml. So some script and doc will need to be adjusted. The warning shows up as:

Warning:jenkins_jobs.builder:(Deprecated)

The default output behavior of jenkins-jobs test when given the --output flag will change in JJB 3.0.
Instead of writing jobs to OUTPUT/jobname; they will be written to OUTPUT/jobname/config.xml.
The new behavior can be enabled by the passing --config-xml parameter.

hashar added a comment.Oct 24 2019, 1:52 PM

2.3.0 has AnsiColor to always set the colormap to xterm.

4cce3d6a78fbe4e1de39299f77acf07943c61a3b

hashar added a comment.Oct 24 2019, 1:55 PM

2.4.0 adds some extra XML parameters for the copyartifact property. That only affects beta-publish-deb:

diff -u output-prev/beta-publish-deb output/beta-publish-deb
--- output-prev/beta-publish-deb	2019-10-24 15:50:08.455101260 +0200
+++ output/beta-publish-deb	2019-10-24 15:53:53.315858163 +0200
@@ -20,6 +20,8 @@
       <optional>false</optional>
       <doNotFingerprintArtifacts>false</doNotFingerprintArtifacts>
       <parameters/>
+      <exclude/>
+      <resultVariableSuffix/>
       <selector class="hudson.plugins.copyartifact.TriggeredBuildSelector">
         <fallbackToLastSuccessful>false</fallbackToLastSuccessful>
       </selector>
hashar added a comment.Oct 24 2019, 2:20 PM

2.9.0 makes the --output related warning to only show once ( e07ad13ad0173a4f5e6ab899cfb6d78861c26d42 ).

gerritbot added a comment.Oct 24 2019, 2:27 PM

Change 545861 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 2.9.1

https://gerrit.wikimedia.org/r/545861

gerritbot added a project: Patch-For-Review.Oct 24 2019, 2:27 PM
gerritbot added a comment.Oct 24 2019, 2:45 PM

Change 545868 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Contrain PyYAML<4

https://gerrit.wikimedia.org/r/545868

hashar added a comment.Oct 24 2019, 3:20 PM

Almost ready to reach latest version available :-]

gerritbot added a comment.Oct 24 2019, 5:29 PM

Change 545868 merged by jenkins-bot:
[integration/config@master] Constrain PyYAML<4

https://gerrit.wikimedia.org/r/545868

gerritbot added a comment.Oct 25 2019, 8:43 AM

Change 546113 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 2.0.9

https://gerrit.wikimedia.org/r/546113

gerritbot added a comment.Oct 25 2019, 8:43 AM

Change 546114 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 2.0.10

https://gerrit.wikimedia.org/r/546114

gerritbot added a comment.Oct 25 2019, 8:43 AM

Change 546115 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 2.2.1

https://gerrit.wikimedia.org/r/546115

gerritbot added a comment.Oct 25 2019, 8:56 AM

Change 545861 abandoned by Hashar:
Upgrade jjb to 2.9.1

Reason:
I am splitting this change.

https://gerrit.wikimedia.org/r/545861

gerritbot added a comment.Oct 25 2019, 9:03 AM

Change 546121 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 2.3.0

https://gerrit.wikimedia.org/r/546121

gerritbot added a comment.Oct 25 2019, 9:10 AM

Change 546130 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 2.9.1

https://gerrit.wikimedia.org/r/546130

gerritbot added a comment.Oct 25 2019, 9:14 AM

Change 546113 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 2.0.9

https://gerrit.wikimedia.org/r/546113

gerritbot added a comment.Oct 25 2019, 9:15 AM

Change 546114 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 2.0.10

https://gerrit.wikimedia.org/r/546114

gerritbot added a comment.Oct 25 2019, 9:17 AM

Change 546115 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 2.2.1

https://gerrit.wikimedia.org/r/546115

gerritbot added a comment.Oct 25 2019, 9:19 AM

Change 546121 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 2.3.0

https://gerrit.wikimedia.org/r/546121

Stashbot added a comment.Oct 25 2019, 9:19 AM

Mentioned in SAL (#wikimedia-releng) [2019-10-25T09:19:29Z] <hashar> Reconfiguring all Jenkins jobs defined by JJB due to upgrade to jjb 2.9.1 # T236036

Stashbot added a comment.Oct 25 2019, 9:20 AM

Mentioned in SAL (#wikimedia-releng) [2019-10-25T09:20:12Z] <hashar> Updated integration/jenkins-job-builder.git to 2.9.1 # T236036

gerritbot added a comment.Oct 25 2019, 9:20 AM

Change 546128 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 2.4.0

https://gerrit.wikimedia.org/r/546128

gerritbot added a comment.Oct 25 2019, 9:20 AM

Change 546130 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 2.9.1

https://gerrit.wikimedia.org/r/546130

hashar closed this task as Resolved.Oct 25 2019, 9:21 AM

Hurrah

hashar edited projects, added Continuous-Integration-Config; removed Patch-For-Review.Oct 25 2019, 9:21 AM
hashar added a subtask: T236685: Migrate JJB usage --config-xml.Oct 28 2019, 4:48 PM
hashar reopened this task as Open.Oct 28 2019, 4:50 PM

I have had an outdated fork on my laptop that was still pointing to an obsolete upstream git. They have since moved to opendev.org which has the tags:

2.9.1   # the one in integration/config
2.10.0
2.10.1
3.0.0
3.0.1
3.0.2

One breaking change is --config-xml which has been solved by T236685: Migrate JJB usage --config-xml

gerritbot added a comment.Oct 28 2019, 4:56 PM

Change 546653 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 2.10.1

https://gerrit.wikimedia.org/r/546653

gerritbot added a project: Patch-For-Review.Oct 28 2019, 4:56 PM
gerritbot added a comment.Oct 28 2019, 5:26 PM

Change 546653 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 2.10.1

https://gerrit.wikimedia.org/r/546653

gerritbot added a comment.Oct 28 2019, 5:39 PM

Change 546663 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Upgrade jjb to 3.0.2

https://gerrit.wikimedia.org/r/546663

gerritbot added a comment.Oct 28 2019, 7:05 PM

Change 546663 merged by jenkins-bot:
[integration/config@master] Upgrade jjb to 3.0.2

https://gerrit.wikimedia.org/r/546663

gerritbot added a comment.Oct 28 2019, 7:43 PM

Change 546698 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Have notice dependency changes

https://gerrit.wikimedia.org/r/546698

hashar added a comment.Oct 28 2019, 7:45 PM

Last thing is that when changing the JJB version in jenkins-job-requirements.txt, tox does not notice the different and would not rebuild the virtual environment. That is tox bug: https://github.com/tox-dev/tox/issues/149

We have to move dependencies and inline them in tox.ini :]

gerritbot added a comment.Oct 28 2019, 8:09 PM

Change 546698 merged by jenkins-bot:
[integration/config@master] Make tox notice dependency changes

https://gerrit.wikimedia.org/r/546698

hashar closed this task as Resolved.Oct 28 2019, 8:20 PM

Thank you @Jdforrester-WMF for the reviews / support etc!

hashar mentioned this in T239232: Archive the integration/jenkins-job-builder repository.Nov 26 2019, 1:26 PM
hashar added a comment.Nov 26 2019, 1:32 PM

And integration/jenkins-job-builder is to be archived with T239232

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL