Page MenuHomePhabricator

Support git-lfs files in gerrit
Closed, ResolvedPublic

Description

WMF currently hosts some stuff in git-fat, but it's optimized for production use. We'd prefer git-lfs since it's compatible with GitHub.

Either way, this task is complete when we can use either flavor of large-file repos from both production and wmflabs deployments.

Details

Related Gerrit Patches:
operations/puppet : productionGerrit: Set auth.gitBasicAuth to true
operations/puppet : productionGerrit: Set lfs configuation

Event Timeline

awight created this task.Jul 26 2017, 4:52 PM
demon added a comment.Jul 26 2017, 4:55 PM

Just wanting to clear up a little bit of FUD, since there seems to be this misconception that git-fat does not work in labs....

This isn't strictly correct. git-fat supports pulling from any place it can hit an rsync daemon. The problem comes in maintaining configuration that's acceptable to both a production deployment as well as labs and a local install. We've hacked around it with testing branches & such for experimentation, but this doesn't scale really and becomes a pain in the ass.

I totally agree with the ideals behind this task: we can and should do better here.

What does *better* look like? What would you do if you had the time to do it?

awight renamed this task from Git-fat should support pulling from production or labs to Simplify git-fat support for pulling from both production and labs.Jul 26 2017, 4:59 PM
awight updated the task description. (Show Details)

@demon Thanks! I've updated the description.

demon added a comment.Jul 26 2017, 5:00 PM

The idea I was noodling the other day was kind of a generic rsync service that "anything" (tbd how wide anything means) could hit. This would allow us to publish files to a single location for all git-fat users rather than having each service having to setup their own (and potentially set up multiple ones for various testing/staging environments as well).

Needs more noodling, but that was just my initial thoughts yesterday or so...

Halfak changed the task status from Open to Stalled.Jul 27 2017, 2:52 PM
Halfak closed this task as Resolved.
Halfak reopened this task as Open.
Halfak raised the priority of this task from Medium to High.
Halfak lowered the priority of this task from High to Medium.
Halfak moved this task from Untriaged to Maintenance/cleanup on the Scoring-platform-team board.
Halfak added a comment.Aug 1 2017, 4:04 PM

Gotcha. Thanks. I think we're interested in putting some energy behind this if you can direct us once your brain-pasta is al dente.

fgiunchedi added a subscriber: fgiunchedi.

CC'ing Operations here too for wider distribution

There was a request in the 2017-08-16 Scrum of Scrums from the Scoring Platform team asking RelEng if this could be prioritized. The task appears to be in "watching" status for RelEng. It sounded like the ball was in @demon's court, but is that true?

demon added a project: Gerrit.EditedAug 18 2017, 9:59 PM

I thought about it yesterday. We should just bite the bullet and get git-lfs support for Gerrit. This is possible and is the best long-term solution. It keeps us from being tied into git-fat (unsupported and effectively forked by us), rsync, or archiva.

There's already git-lfs plugins for Gerrit--fs-based and s3-based. S3 serves as the model we'd follow.

demon moved this task from Bugs & stuff to Needs a plugin! on the Gerrit board.Aug 18 2017, 9:59 PM

/me likes @demon's post.

Awesome, let's stay in coordination about how we might be able to help with this effort. Seems that there's some reasonably active code to get us started:
https://gerrit.googlesource.com/plugins/lfs/

awight renamed this task from Simplify git-fat support for pulling from both production and labs to Support git-lfs files in gerrit.Sep 9 2017, 12:54 AM
awight updated the task description. (Show Details)

Adding our Release-Engineering-Team (Kanban) project as we would like to work on this in the coming quarter or two (no promises though, this is not a "goal" only "other hoped for work").

mmodell added a subscriber: mmodell.EditedOct 12 2017, 12:29 AM

Relatedly, phabricator upstream has rudimentary git-lfs support, sort of documented in the comments on https://secure.phabricator.com/T7789. If we move phabricator's storage engine to swift we could use it as a universal git large file repository.

There’s a 2.13 branch https://gerrit.googlesource.com/plugins/lfs/+/stable-2.13

Docs is at https://gerrit.googlesource.com/plugins/lfs/+/stable-2.13/src/main/resources/Documentation/config.md

All we need to do is install the plugin and configure the settings to match our needs. It defaults to the local file system so that’s ok :).

We could possibly do FS-based for now, pending work on supporting Swift.

demon added a comment.EditedOct 14 2017, 1:01 AM

The /srv partition on Gerrit is only 1% full, we've got plenty of space.

Change 384596 had a related patch set (by Paladox) published:
[operations/puppet@production] Gerrit: Set lfs configuation

https://gerrit.wikimedia.org/r/384596

I've done the configuration. We need to install the plugin. But the good thing is we can install things, and if the configuration is wrong, we can disable the plugin through gerrit's ui and then fix the configuration.

also

lfs.url needs to be set in each plugin's project.config that will be using lfs.

lfs.url : http://<username>@<gerrit-host>:<port>/<project-name>/info/lfs

ie something like ^^ or changed to ssh format.

Change 384596 merged by Dzahn:
[operations/puppet@production] Gerrit: Set lfs configuation

https://gerrit.wikimedia.org/r/384596

Change 385239 had a related patch set uploaded (by Paladox; owner: Paladox):
[All-Projects@refs/meta/config] Enable lfs for mediawiki/services/chromium-render/deploy

https://gerrit.wikimedia.org/r/385239

I have tested the plugin. The configuration above should work ^^ :).

then we have to initialise the repo with git lfs install.

Then do https://help.github.com/articles/configuring-git-large-file-storage/

phuedx added a subscriber: phuedx.Nov 3 2017, 6:00 PM

LFS is now supported and enabled on a testing repo.

As i found out today, lfs dosent seem to support digest which is enabled by default for gerrit 2.13 but removed in 2.14+. Switching auth.gitBasicAuth to true will enable basic auth and disable digest.

Change 391865 had a related patch set uploaded (by Paladox; owner: Paladox):
[operations/puppet@production] Gerrit: Set auth.gitBasicAuth to true

https://gerrit.wikimedia.org/r/391865

Change 391865 merged by Dzahn:
[operations/puppet@production] Gerrit: Set auth.gitBasicAuth to true

https://gerrit.wikimedia.org/r/391865

Mentioned in SAL (#wikimedia-operations) [2017-11-29T19:55:12Z] <mutante> restarting gerrit to apply config change and set gitBasicAuth to true to unblock T171758 (gerrit:391865)

demon closed this task as Resolved.Nov 29 2017, 8:31 PM
demon claimed this task.