WMF currently hosts some stuff in git-fat, but it's optimized for production use. We'd prefer git-lfs since it's compatible with GitHub.
Either way, this task is complete when we can use either flavor of large-file repos from both production and wmflabs deployments.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Gerrit: Set auth.gitBasicAuth to true | operations/puppet | production | +1 -0 | |
| Gerrit: Set lfs configuation | operations/puppet | production | +28 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Halfak | T171619 [Epic] ORES should use a git large file plugin for storing serialized binaries | |||
| Resolved | • demon | T171758 Support git-lfs files in gerrit | |||
| Declined | None | T179009 ORES deployment submodules should point to phabricator HTTPS repos. |
Just wanting to clear up a little bit of FUD, since there seems to be this misconception that git-fat does not work in labs....
This isn't strictly correct. git-fat supports pulling from any place it can hit an rsync daemon. The problem comes in maintaining configuration that's acceptable to both a production deployment as well as labs and a local install. We've hacked around it with testing branches & such for experimentation, but this doesn't scale really and becomes a pain in the ass.
I totally agree with the ideals behind this task: we can and should do better here.
The idea I was noodling the other day was kind of a generic rsync service that "anything" (tbd how wide anything means) could hit. This would allow us to publish files to a single location for all git-fat users rather than having each service having to setup their own (and potentially set up multiple ones for various testing/staging environments as well).
Needs more noodling, but that was just my initial thoughts yesterday or so...
Gotcha. Thanks. I think we're interested in putting some energy behind this if you can direct us once your brain-pasta is al dente.
There was a request in the 2017-08-16 Scrum of Scrums from the Scoring Platform team asking RelEng if this could be prioritized. The task appears to be in "watching" status for RelEng. It sounded like the ball was in @demon's court, but is that true?
I thought about it yesterday. We should just bite the bullet and get git-lfs support for Gerrit. This is possible and is the best long-term solution. It keeps us from being tied into git-fat (unsupported and effectively forked by us), rsync, or archiva.
There's already git-lfs plugins for Gerrit--fs-based and s3-based. S3 serves as the model we'd follow.
/me likes @demon's post.
Awesome, let's stay in coordination about how we might be able to help with this effort. Seems that there's some reasonably active code to get us started:
https://gerrit.googlesource.com/plugins/lfs/
Adding our Release-Engineering-Team (Kanban) project as we would like to work on this in the coming quarter or two (no promises though, this is not a "goal" only "other hoped for work").
Relatedly, phabricator upstream has rudimentary git-lfs support, sort of documented in the comments on https://secure.phabricator.com/T7789. If we move phabricator's storage engine to swift we could use it as a universal git large file repository.
There’s a 2.13 branch https://gerrit.googlesource.com/plugins/lfs/+/stable-2.13
Docs is at https://gerrit.googlesource.com/plugins/lfs/+/stable-2.13/src/main/resources/Documentation/config.md
All we need to do is install the plugin and configure the settings to match our needs. It defaults to the local file system so that’s ok :).
Change 384596 had a related patch set (by Paladox) published:
[operations/puppet@production] Gerrit: Set lfs configuation
I've done the configuration. We need to install the plugin. But the good thing is we can install things, and if the configuration is wrong, we can disable the plugin through gerrit's ui and then fix the configuration.
also
lfs.url needs to be set in each plugin's project.config that will be using lfs.
lfs.url : http://<username>@<gerrit-host>:<port>/<project-name>/info/lfs
ie something like ^^ or changed to ssh format.
Change 384596 merged by Dzahn:
[operations/puppet@production] Gerrit: Set lfs configuation
Change 385239 had a related patch set uploaded (by Paladox; owner: Paladox):
[All-Projects@refs/meta/config] Enable lfs for mediawiki/services/chromium-render/deploy
I have tested the plugin. The configuration above should work ^^ :).
then we have to initialise the repo with git lfs install.
Then do https://help.github.com/articles/configuring-git-large-file-storage/
LFS is now supported and enabled on a testing repo.
As i found out today, lfs dosent seem to support digest which is enabled by default for gerrit 2.13 but removed in 2.14+. Switching auth.gitBasicAuth to true will enable basic auth and disable digest.
Change 391865 had a related patch set uploaded (by Paladox; owner: Paladox):
[operations/puppet@production] Gerrit: Set auth.gitBasicAuth to true
Change 391865 merged by Dzahn:
[operations/puppet@production] Gerrit: Set auth.gitBasicAuth to true
Mentioned in SAL (#wikimedia-operations) [2017-11-29T19:55:12Z] <mutante> restarting gerrit to apply config change and set gitBasicAuth to true to unblock T171758 (gerrit:391865)