There's debian packages for git-lfs, but they're binary packages so we'll probably want/need to build it ourselves (hopefully not hard, they did the work).
If it's not too hard, we should just install it everywhere we install git, just for consistency. But at the very least it should be installed on all scap-related hosts (masters & targets)
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | awight | T187217 [Epic] Support word2vec for production ORES models | |||
| Resolved | awight | T188446 Package word2vec binaries | |||
| Resolved | Halfak | T171619 [Epic] ORES should use a git large file plugin for storing serialized binaries | |||
| Resolved | awight | T181678 Plan migration of ORES repos to git-lfs | |||
| Resolved | None | T176324 Scoring platform team FY18 Q2 | |||
| Resolved | Halfak | T183198 Scoring Platform FY18 Q3 | |||
| Resolved | awight | T176336 Deploy drafttopic model to production ORES | |||
| Resolved | • mmodell | T180627 Support git-lfs in scap | |||
| Resolved | awight | T181010 [Spike] Write reports about why Ext:ORES is helping cause server 500s and write tasks to fix | |||
| Resolved | awight | T181071 Cache ORES virtualenv within versioned source | |||
| Resolved | fgiunchedi | T189306 SCAP: Upload debian package version 3.7.7-1 | |||
| Resolved | awight | T180628 Install git-lfs client (at least on scap targets & masters) |
@thcipriani Noticed that it appears it's been added to buster. So should just be a backport?
@akosiaris: What would it take to get the git-lfs package back-ported to stretch? It's written in go, however, I am unsure if it will work with the version of go that's available in stretch.
A direct backporting and rebuilding is impossible as it turns out. There just way too many dependencies that are present in buster and not in stretch. That being said, it's a go app so it compiles to not really dynamically linked executable (aside from libpthread and libc) so the same package provided for buster works on stretch as well (and I did a quick test of git lfs install and git lfs track to verify).
I 'll try and have it uploaded to stretch-wikimedia by the end of the day.
Change 417226 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] scap::target: Install git-lfs
Change 417226 merged by Alexandros Kosiaris:
[operations/puppet@production] scap::target: Install git-lfs
The scap targets that would benefit from this (namely ores* boxes) now have git-lfs installed. @mmodell do we also need this on the scap masters ? I am not fully clear about the workflow that is going to be used here and where the git lfs related files are going to be fetched from.
@akosiaris: I think it's needed on masters, at least to enable deployers to issue git-lfs commands. I'm unsure if scap itself does git-lfs commands on the master, I believe it's only on targets.
Why do the deployers issue git-lfs commands on the masters? Is it required per the process ? or just beneficial ? or nothing of the above ? Should we jot down the workflow (and even get it working in beta or some other environment) before we start trying it in production ?
@akosiaris: tl;dr I can't think of any reason that we must have git-lfs on masters, I've only got vague hand-wavy notions about it being convenient.
I don't know if the workflow requires it, I was really just imagining that it'd be convenient for debugging. Now that you mention it, I agree, it would be a good idea to figure out the workflow and get it written down somewhere. Trying it on beta is probably a good idea as well.
I've double-checked and it appears that we are only calling git-lfs/git-fat from one place in deploy-local, so git-lfs definitely isn't required (by scap) to be installed on masters.
Maybe @demon or @awight have a better idea than I do about the workflow. I'm only just getting familiar with git-lfs myself and I have never used git-fat (which is essentially the same workflow, afaik)..
I can't think of any compelling reason why it would be *required* on the masters...but I could see it as being useful if you want to actually have your large file contents on the master if you're debugging/curious about something...?
Change 420409 had a related patch set uploaded (by Awight; owner: Awight):
[operations/puppet@production] Install git-lfs on scap source and target
The catch seems to be that tin and deployment-tin are running jessie, so the git-lfs package isn't easily available.
tin has been replaced by deploy1001 which is running stretch
[deploy1001:~] $ apt-cache show git-lfs
Package: git-lfs
Version: 2.3.4-1
(not installed yet though)
Change 420409 merged by Alexandros Kosiaris:
[operations/puppet@production] Install git-lfs on scap source and target
Confirmed that git-lfs is installed on deploy1001, removing blocking task dependency.