HomePhabricator
Diffusion Scap ccea24641f77

Cache submodules and use --reference to save space

Description

Cache submodules and use --reference to save space

Summary:
Requires git 2.11, which we should have everywhere.

The new behavior is to caches the submodules in deploy-cache/cache/modules/, then
when cloning to revs/$rev/ we use --recurse-submodules and --reference ../cache/
then git does the magic to make the clone's submodules reuse the cached objects.

Disk usage, using rPHDEP as an example.

Cache modules

$ du -hs cache/.git/modules/
121M    cache/.git/modules/

Checkout in revs/

$ du -hs revs/test/..git/modules
2.6M    revs/test/.git/modules

Test Plan: Currently untested. I'd like to merge this and test in beta.

Reviewers: demon, thcipriani, hashar, dduvall, Release-Engineering-Team

Reviewed By: demon, Release-Engineering-Team

Tags: Release-Engineering-Team

Maniphest Tasks: T137124

Differential Revision: https://phabricator.wikimedia.org/D826

Details

Provenance
mmodellAuthored on Oct 18 2017, 10:16 PM
Reviewer
Release-Engineering-Team
Differential Revision
D826: Cache submodules and use --reference to save space
Parents
rMSCA2561b82f6211: Unify "give me some number of CPUs I can use for subtasks" logic
Branches
Unknown
Tags
Unknown
ChangeId
None
Build Status
Buildable 2387
Build 3896: ci-jessieJenkins