Page MenuHomePhabricator

scap support for git-lfs
Closed, DuplicatePublic

Description

What will happen if we try to checkout a project with git-lfs-enabled submodules on tin?
And will scap be able to fetch and checkout on deployment targets?

Event Timeline

awight created this task.Dec 1 2017, 9:16 PM
demon added a comment.Dec 4 2017, 5:46 PM

What will happen if we try to checkout a project with git-lfs-enabled submodules on tin?

Scap does not speak git-lfs yet, but this shouldn't be too difficult. It'd basically be a few tweaks where we currently have git-fat support.

And will scap be able to fetch and checkout on deployment targets?

See above. The only caveat is that I'm a little concerned about having all the targets hammer Gerrit or Phab all at once for multi-GB files to stream.... we'll need some further thoughts here :\

awight added a comment.Dec 4 2017, 5:51 PM

And will scap be able to fetch and checkout on deployment targets?

See above. The only caveat is that I'm a little concerned about having all the targets hammer Gerrit or Phab all at once for multi-GB files to stream.... we'll need some further thoughts here :\

This is naive, I don't have any of the necessary details here, but maybe git-lfs should be behind a caching proxy?

demon added a comment.Dec 4 2017, 6:01 PM

Phab's already behind Varnish, Gerrit is not yet (cf some bug I don't have in front of me). But with objects this large can we actually expect Varnish to keep them in memory?

awight added a comment.Dec 4 2017, 6:04 PM

Good point, that won't work at all.

On the bright side, we know that the git-lfs load is actually smaller than the current load (since we're no longer pulling the repo history), so if we're pointing at the same machines as before then this shouldn't be an immediate blocker. In the long run, we'll want a better answer of course.

demon added a comment.Dec 4 2017, 6:13 PM

Yep I think we're on the same page here.

Ottomata triaged this task as Medium priority.Jan 16 2018, 7:39 PM
Ottomata added a subscriber: Ottomata.

Just curious, why not use git fat? We have a git-fat store available already, and it can be used by scap:

https://wikitech.wikimedia.org/wiki/Archiva#Setting_up_git-fat_for_your_project

The git-fat store we set up was fro deploying Java artifacts (.jars), but it can be used for anything. We could move this to a more generic server(s) if we wanted to.

demon added a comment.Jan 16 2018, 8:00 PM

Just curious, why not use git fat? We have a git-fat store available already, and it can be used by scap:
https://wikitech.wikimedia.org/wiki/Archiva#Setting_up_git-fat_for_your_project
The git-fat store we set up was fro deploying Java artifacts (.jars), but it can be used for anything. We could move this to a more generic server(s) if we wanted to.

Anything that speaks rsync, sure--the overhead for setting it up is a little more difficult. I think part of the motivation is that we've got repos we also want to operate with outside of WMF (like Gitlab). git-lfs has become the de facto standard in the industry here so there's no harm in supporting it. It doesn't preclude us from also using git-fat (see D921, where I introduce support and in a way that retains git-fat as well)

K cool, sounds good :)

mmodell moved this task from Needs triage to Debt on the Scap board.Feb 1 2018, 12:22 AM
awight added a comment.Jun 4 2018, 9:29 PM
This comment was removed by awight.