Page MenuHomePhabricator

LFS objects are not mirroring from Github through Phab to Gerrit consistently
Closed, DeclinedPublic

Description

Here's an example failure: T212530#4850623

Gerrit mirrors that use LFS are not pulling git lfs files. A work-around is necessary for sending these files through LFS. See T212818#4865068. This workaround should be unnecessary. Gerrit should do the git lfs fetch for us.

This task is done when: a user can push LFS files to a source git repo and expect that a gerrit mirror of that repo will pull (mirror) the LFS files.

Event Timeline

Hi! This seems to be a much different issue than the articlequality glitch, so I'm reopening.

Can you better describe the issue? It is not clear to me what is happening here.

This issue is easily fixable. Let me explain why it happens. (Probably duplicate of T212544). The problem happens only when there is new git lfs file added.

In git lfs, tracked files are replaced with a text files that points out to the real file. For example in editquality the models/frwiki.damaging.gradient_boosting.model file is just:

version https://git-lfs.github.com/spec/v1
oid sha256:c9534ef5f4157659cff69b676829e2e049bc9b25604102315fb18bf15d13252f
size 4577139

So doing git pull just grabs these text files and then doing git lfs pull makes git to loads this file from the LFS storage endpoint of github (It looks it based on the hash mentioned in the pointer)

editquality repo is getting mirrored from https://github.com/wikimedia/editquality to https://gerrit.wikimedia.org/r/scoring/ores/editquality (gerrit one is also another mirror but that's another topic and doesn't matter here).

So the pointer files gets mirrored but the actual files are not being mirrored (also because all of them live in a central storage service and mirroring all of github lfs storage doesn't sound feasible)

But people with +2 rights actually have lfs push rights (it's not the same as git push). What git lfs push does is that it uploads the real files in your local machine to the storage. So the way that this can be bypassed is to simply upload the files from the actual place. Let me demonstrate it by this:

$ git clone https://github.com/wikimedia/editquality
$ cd editquality
$ git lfs pull
$ git remote add gerrit https://gerrit.wikimedia.org/r/scoring/ores/editquality
$ git lfs push gerrit master

It would upload all of the files to the gerrit lfs storage (I think it's phabricator but that doesn't matter)

Then if you do:

$ git clone https://gerrit.wikimedia.org/r/scoring/ores/editquality editquality-gerrit
$ cd editquality-gerrit
$ git lfs pull

No issue would happen.

Halfak claimed this task.
Halfak triaged this task as Medium priority.
Halfak moved this task from Unsorted to Maintenance/cleanup on the Machine-Learning-Team board.
Halfak added a subscriber: Halfak.

It looks like @Ladsgroup's hack has resolved this. So I'm closing. Feel free to re-open.

I ran into this issue again. I needed to do @Ladsgroup's work around. It looks like LFS is still not automatically mirrored to gerrit.

bd808 removed Halfak as the assignee of this task.Feb 5 2021, 10:34 PM
elukey added a subscriber: elukey.

This is now documented on Wikitech, and there seems to be no long term plan to fix this (Lift Wing will use Swift to store models). I am inclined to just close the task.