Diffusion Scap ccea24641f77

Cache submodules and use --reference to save space

Authored by mmodell on Oct 18 2017, 10:16 PM.


Cache submodules and use --reference to save space

Requires git 2.11, which we should have everywhere.

The new behavior is to caches the submodules in deploy-cache/cache/modules/, then
when cloning to revs/$rev/ we use --recurse-submodules and --reference ../cache/
then git does the magic to make the clone's submodules reuse the cached objects.

Disk usage, using rPHDEP as an example.

Cache modules

$ du -hs cache/.git/modules/
121M    cache/.git/modules/

Checkout in revs/

$ du -hs revs/test/..git/modules
2.6M    revs/test/.git/modules

Test Plan: Currently untested. I'd like to merge this and test in beta.

Reviewers: demon, thcipriani, hashar, dduvall, Release-Engineering-Team

Reviewed By: demon, Release-Engineering-Team

Tags: Release-Engineering-Team

Maniphest Tasks: T137124

Differential Revision: https://phabricator.wikimedia.org/D826