When checking out projects for test, zuul-cloner should also check out their submodules. For example, this DonationInterface change can't merge because the vendor submodule is not checked out and phpunit cannot find autoloader.php: https://gerrit.wikimedia.org/r/180894
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | PRODUCTION ERROR | hashar | T155773 LinkSuggest2 test failing due to missing files located in sub repo | ||
| Resolved | PRODUCTION ERROR | hashar | T154930 PagesList tests failing due to missing files located in sub repo | ||
| Resolved | PRODUCTION ERROR | Paladox | T154848 GooglePlaces tests failing due to missing files located in sub repo | ||
| Resolved | PRODUCTION ERROR | Paladox | T154847 FlickrAPI test failing due to missing files located in sub repo | ||
| Resolved | PRODUCTION ERROR | hashar | T154806 Skin Refreshed sub repo does not handled in test config | ||
| Resolved | PRODUCTION ERROR | hashar | T154935 PaginateText extension: Tests failing due to missing files located in sub repo | ||
| Resolved | hashar | T130966 generalize extension submodule handling | |||
| Resolved | awight | T86267 DonationInterface should autoload 3rd party dependencies via Composer | |||
| Resolved | Ejegg | T89404 Create unit and integration tests for Fundraising extensions to identify breaking MediaWiki changes | |||
| Declined | hashar | T84942 Zuul-cloner should check out submodules |
Event Timeline
Comment Actions
Unless the submodules use change a lot, this should not be a blocker for other tasks. We already have a handful of projects using submodules which are fetched with a custom shell macro in the relevant Jenkins jobs.
Note that adding such a submodule macro, however, significantly slows down the job due to git having to fetch (clone) the entire project from gerrit.wikimedia.org (as specified in .gitmodules). In contrary to the main repo, which is fetched from the zuul-merger and also cached between builds.
Comment Actions
So, I realized that there is no workaround for this issue. Consider these two scenarios:
- Parent and submodule repos are checked out using zuul-clone. The benefit is that the job can be reused for submodules, because the ZUUL environment variables will cause the correct submodule change to be checked out. The drawback is that a parent change which includes submodule bumps will be tested with the submodules at their master revision.
- Integration script includes a "git submodule update" or similar. The benefit is that submodule bumps are correctly tested along with the parent patch. The drawback is that the job cannot be reused for submodule repos.
This is currently an issue for wikimedia/fundraising/crm and its submodules.
Comment Actions
The logic to do this correctly seems pretty straightforward, my only question is what the clone statement would look like for submodule projects that should be tested as a component of their parent repo. I would be happy with something like this, https://gerrit.wikimedia.org/r/#/c/195063/11/jjb/wm-fundraising.yaml,unified
In that case, an effective but expensive algorithm would be,
- Clone the top level repo and checkout all submodules at the revision specified.
- Parse zuul-project to determine the directory and repo of the current change.
- Re-checkout that directory if its SHA1 doesn't match zuul-change.
A more optimized algorithm would have to intervene before git submodule update. It would go something like,
- Do a shallow zuul-clone of the top level repo.
- Traverse the submodule tree, doing a shallow zuul clone at each node.