Should be doable by using zuul-cloner. The utility clones multiple repositories then checkout branches/patches based on some conventions or an indicated branch to use. Upstream doc: http://docs.openstack.org/infra/zuul/cloner.html#usage

In mediawiki/core@fundraising/REL1_25 we have the following submodules:

[submodule "extensions/DonationInterface"]
[submodule "extensions/ContributionTracking"]
[submodule "extensions/ParserFunctions"]
[submodule "extensions/DonationEmailUnsubscribe"]
[submodule "extensions/cldr"]
[submodule "skins/Vector"]
[submodule "vendor"]

So at first I guess we will want a job that runs whenever a patch is proposed for mediawiki/core@fundraising/REL1_25 which will process submodules then run the extensions tests.

Second, to catch issues before the submodule bumps is proposed, we will want a job that run on all those extensions against whatever branch you use. Given an extension is developed on master, zuul-cloner automatically match mediawiki/core @ master, when I guess you will want fundraising/REL1_25.

The branch can be overridden in zuul-cloner on a per project basis, the syntax would be: --project-branch mediawiki/core=fundraising/REL1_25.

The command line would then be something like:

zuul-cloner
  --branch master
  --project-branch mediawiki/core=fundraising/REL1_25
  extensions/DonationInterface
  extensions/ContributionTracking
  extensions/ParserFunctions
  extensions/DonationEmailUnsubscribe
  extensions/cldr
  skins/Vector
  vendor

The job would be triggered on changes proposed to master branches of all the repositories (except mediawiki/core). It will then clone:

• the repo that triggered the change checked out to the patch
• the master branches of the other repos
• fundraising/REL1_25 branch of mediawiki/core

Change 266653 had a related patch set uploaded (by Cdentinger):
WIP: Adds DonationInterface test against mediawiki 1.25

https://gerrit.wikimedia.org/r/266653

@hashar this is very preliminary but the above edits seem to produce more or less the Jenkins job that I think we need. Please advise/correct, whatever you have time for!

@cwdent that looks awesome! Will follow up on the Gerrit change.

@hashar thank you! I really appreciate the help. I made those couple edits as well as some other cleanup. I also got the Jenkins API stuff set up so if you are OK with it I will start updating the job and testing.

@hashar Ok, the latest PS passes: https://gerrit.wikimedia.org/r/#/c/266653

I used --branch fundraising/REL1_25 which pulls the correct thing for core and core/vendor, the two repos where we are pinned to substantially different revisions than master. Then I used --project-branch to set DonationInterface to master, what we want to test.

One problem (that I know of) remains, and I'm wondering if it has to do with selecting branches in a different way than the other jobs. If the build fails for any reason, the next build will fail with these errors:

23:22:23 INFO:zuul.Cloner:Creating repo mediawiki/core from upstream https://gerrit.wikimedia.org/r/p/mediawiki/core
23:22:23 DEBUG:zuul.Repo:Cloning from https://gerrit.wikimedia.org/r/p/mediawiki/core to src
23:22:23 ERROR:zuul.Repo:Unable to initialize repo for https://gerrit.wikimedia.org/r/p/mediawiki/core
23:22:23 Traceback (most recent call last):
23:22:23   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 53, in __init__
23:22:23     self._ensure_cloned()
23:22:23   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/zuul/merger/merger.py", line 65, in _ensure_cloned
23:22:23     git.Repo.clone_from(self.remote_url, self.local_path)
23:22:23   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/repo/base.py", line 849, in clone_from
23:22:23     return cls._clone(Git(os.getcwd()), url, to_path, GitCmdObjectDB, progress, **kwargs)
23:22:23   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/repo/base.py", line 800, in _clone
23:22:23     finalize_process(proc)
23:22:23   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/util.py", line 154, in finalize_process
23:22:23     proc.wait()
23:22:23   File "/usr/share/python/zuul/local/lib/python2.7/site-packages/git/cmd.py", line 309, in wait
23:22:23     raise GitCommandError(self.args, status, self.proc.stderr.read())
23:22:23 GitCommandError: 'git clone -v https://gerrit.wikimedia.org/r/p/mediawiki/core src' returned with exit code 128
23:22:23 stderr: 'fatal: destination path 'src' already exists and is not an empty directory.
23:22:23 '

That will happen until the src dir is removed by hand. My understanding is the job should re-use that dir to avoid wasting time with fresh clones, so I don't understand why that directory being present causes a problem. I was considering making a publisher to rm -rf, but wanted to check with you first in case there is a known issue.

Change 267690 had a related patch set uploaded (by Cdentinger):
WIP add new DonationInterface test to layout.yaml

https://gerrit.wikimedia.org/r/267690

@hashar I just pushed up my first guess at what the changes to layout.yaml should look like: https://gerrit.wikimedia.org/r/#/c/267690/

Change 266653 merged by jenkins-bot:
Adds DonationInterface test against mediawiki 1.25

https://gerrit.wikimedia.org/r/266653

Change 267690 merged by jenkins-bot:
add new DI test to experimental pipeline

https://gerrit.wikimedia.org/r/267690

Change 268021 had a related patch set uploaded (by Cdentinger):
Make new DI job live, set old one to non-voting

https://gerrit.wikimedia.org/r/268021

Change 268021 merged by jenkins-bot:
Make new DI job live, set old one to non-voting

https://gerrit.wikimedia.org/r/268021