I've created a draft implementation for this: https://gitlab.wikimedia.org/repos/test-platform/catalyst/ci-charts/-/merge_requests/44. Using it locally I can see a clear improvement in env time provisioning. For example, for envs created using the Wikipedia preset:

• With ref clones: 6m 27s
• With workspaces: 4m 41s

Unfortunately, the solution above requires mounting the repo pool in write mode, since the medatada about workspaces gets stored in the pool repos. This is less than ideal, as it breaks isolation between the components using the pool. In particular the ci-charts/mediawiki chart could add modifications to repos in the pool that would affect other Catalyst charts, as well as Patchdemo. This may still be an acceptable solution though, because we control ci-charts/mediawiki.

I can think of three immediate options to move forward:

1. We go with the solution in my draft, i.e. we allow ci-charts/mediawiki to modify the pool:
   • PROS: Simple. Not dangerous as long as we remember what the chart is doing
   • CONS: Flawed design. Introduces a more-or-less-hidden coupling between charts and clients of the pool (e.g. Patchdemo) that might come back to bite us down the road
2. We create a separate, dedicated repo pool for ci-charts/mediawiki. This new pool could be created and managed by Catalyst and based on the existing pool
   • PROS: Properly isolates all charts used by Catalyst and other clients of the pool
   • CONS: Not very hard to implement, but it's added complexity nonetheless. Envs created with ci-charts/mediawiki can theoretically still affect each other
3. We do nothing. We ignore the cloning times and focus our efforts on improving other parts of the provisioning process, i.e. npm and composer
   • PROS: Done! :)
   • CONS: Cloning times still sad

After some discussion, we decided to go with option 1.: We'll use worktrees.

The idea seemed reasonable to everyone and nobody was particularly concerned about accidental modifications to the pool in the future

jnuche updated https://gitlab.wikimedia.org/repos/test-platform/catalyst/ci-charts/-/merge_requests/44

use git worktree to create checkouts of the MW repos

jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/101

remove repository pool path from calls to Catalyst API

jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-api/-/merge_requests/87

migrate default repo cache configuration for the mediawiki chart

jnuche opened https://gitlab.wikimedia.org/repos/test-platform/catalyst/wiki-repos-pool/-/merge_requests/8

do not chown the pool repos to user 33 (www-data)

jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/wiki-repos-pool/-/merge_requests/8

do not chown the pool repos to user 33 (www-data)

jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-api/-/merge_requests/87

migrate default repo cache configuration for the mediawiki chart

jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/101

remove repository pool path from calls to Catalyst API

jnuche merged https://gitlab.wikimedia.org/repos/test-platform/catalyst/ci-charts/-/merge_requests/44

use git worktree to create checkouts of the MW repos

@thcipriani and I briefly talked about this task.

• A clone by reference sets a .git/objects/info/alternates that contains the path to the directory holding objects, it then retrieve this objects to create the files.
• Worktree has the git directory under the original git repo (under its .git/worktrees/hotfix), files are copied from it to create the files.

So essentially they should be as fast since, beside the few .git files copied, they are doing essentially the same operation: create directories, create the files, done.

In both cases the cache is updated from Gerrit, but git clone does a second fetch to get the live objects. On my setup the fetch takes some seconds and the difference is thus noticeable. I think that explains the bulk of it.

Another thing I have noticed with strace -f -c is that worktree does less system calls 68774 versus 77142 for a git clone by reference. Most of the extra ones are a lot more newfstatat (24k > 32k) and more read (210 > 1975). So I guess if the disk I/O are slow (as is the case on WMCS), that adds to the slowness.

Regardless git-worktree looks nice, and is faster in the end. Well done!