Page MenuHomePhabricator

Zuul-cloner should check out submodules
Closed, DeclinedPublic


When checking out projects for test, zuul-cloner should also check out their submodules. For example, this DonationInterface change can't merge because the vendor submodule is not checked out and phpunit cannot find autoloader.php:

Event Timeline

Ejegg assigned this task to hashar.
Ejegg raised the priority of this task from to Needs Triage.
Ejegg updated the task description. (Show Details)
Ejegg added a project: Zuul.
Ejegg changed Security from none to None.
Ejegg added subscribers: Ejegg, awight.

Unless the submodules use change a lot, this should not be a blocker for other tasks. We already have a handful of projects using submodules which are fetched with a custom shell macro in the relevant Jenkins jobs.

Note that adding such a submodule macro, however, significantly slows down the job due to git having to fetch (clone) the entire project from (as specified in .gitmodules). In contrary to the main repo, which is fetched from the zuul-merger and also cached between builds.

So, I realized that there is no workaround for this issue. Consider these two scenarios:

  1. Parent and submodule repos are checked out using zuul-clone. The benefit is that the job can be reused for submodules, because the ZUUL environment variables will cause the correct submodule change to be checked out. The drawback is that a parent change which includes submodule bumps will be tested with the submodules at their master revision.
  2. Integration script includes a "git submodule update" or similar. The benefit is that submodule bumps are correctly tested along with the parent patch. The drawback is that the job cannot be reused for submodule repos.

This is currently an issue for wikimedia/fundraising/crm and its submodules.

The logic to do this correctly seems pretty straightforward, my only question is what the clone statement would look like for submodule projects that should be tested as a component of their parent repo. I would be happy with something like this,,unified

In that case, an effective but expensive algorithm would be,

  • Clone the top level repo and checkout all submodules at the revision specified.
  • Parse zuul-project to determine the directory and repo of the current change.
  • Re-checkout that directory if its SHA1 doesn't match zuul-change.

A more optimized algorithm would have to intervene before git submodule update. It would go something like,

  • Do a shallow zuul-clone of the top level repo.
  • Traverse the submodule tree, doing a shallow zuul clone at each node.