Page MenuHomePhabricator

Gerrit repo scoring/ores/editquality LFS broken (smudge filter lfs failed)
Closed, ResolvedPublic

Description

$ git clone https://gerrit.wikimedia.org/r/scoring/ores/editquality eq_gerrit
Cloning into 'eq_gerrit'...
remote: Counting objects: 3778, done
remote: Finding sources: 100% (4661/4661)
remote: Getting sizes: 100% (1216/1216)
remote: Compressing objects: 100% (12246/12246)
remote: Total 4661 (delta 2456), reused 4654 (delta 2452)
Receiving objects: 100% (4661/4661), 3.87 MiB | 0 bytes/s, done.
Resolving deltas: 100% (2456/2456), done.
Checking connectivity... done.
Downloading models/arwiki.damaging.gradient_boosting.model (360 KB)
Error downloading object: models/arwiki.damaging.gradient_boosting.model (7b868ea): Smudge error: Error downloading models/arwiki.damaging.gradient_boosting.model (7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a): [7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a] Object '7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a' not found: [404] Object '7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a' not found

Errors logged to /home/halfak/projects/eq_gerrit/.git/lfs/logs/20181221T173440.706423731.log
Use `git lfs logs last` to view the log.
error: external filter git-lfs smudge -- %f failed 2
error: external filter git-lfs smudge -- %f failed
fatal: models/arwiki.damaging.gradient_boosting.model: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

Event Timeline

Here's the error log:

git-lfs/2.6.0 (GitHub; linux amd64; go 1.11.1)
git version 2.7.4

$ git-lfs smudge -- models/arwiki.damaging.gradient_boosting.model
Error downloading object: models/arwiki.damaging.gradient_boosting.model (7b868ea): Smudge error: Error downloading models/arwiki.damaging.gradient_boosting.model (7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a): [7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a] Object '7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a' not found: [404] Object '7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a' not found

[404] Object '7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a' not found
[7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a] Object '7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a' not found
github.com/git-lfs/git-lfs/errors.newWrappedError
	/tmp/docker_run/src/git-lfs/amd64/errors/types.go:184
github.com/git-lfs/git-lfs/errors.Wrapf
	/tmp/docker_run/src/git-lfs/amd64/errors/errors.go:85
github.com/git-lfs/git-lfs/tq.(*TransferQueue).enqueueAndCollectRetriesFor
	/tmp/docker_run/src/git-lfs/amd64/tq/transfer_queue.go:470
github.com/git-lfs/git-lfs/tq.(*TransferQueue).collectBatches.func1
	/tmp/docker_run/src/git-lfs/amd64/tq/transfer_queue.go:361
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1333
Error downloading models/arwiki.damaging.gradient_boosting.model (7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a)
github.com/git-lfs/git-lfs/errors.newWrappedError
	/tmp/docker_run/src/git-lfs/amd64/errors/types.go:184
github.com/git-lfs/git-lfs/errors.Wrapf
	/tmp/docker_run/src/git-lfs/amd64/errors/errors.go:85
github.com/git-lfs/git-lfs/lfs.(*GitFilter).downloadFile
	/tmp/docker_run/src/git-lfs/amd64/lfs/gitfilter_smudge.go:110
github.com/git-lfs/git-lfs/lfs.(*GitFilter).Smudge
	/tmp/docker_run/src/git-lfs/amd64/lfs/gitfilter_smudge.go:71
github.com/git-lfs/git-lfs/commands.smudge
	/tmp/docker_run/src/git-lfs/amd64/commands/command_smudge.go:127
github.com/git-lfs/git-lfs/commands.smudgeCommand
	/tmp/docker_run/src/git-lfs/amd64/commands/command_smudge.go:161
github.com/spf13/cobra.(*Command).execute
	/tmp/docker_run/src/git-lfs/amd64/obj-x86_64-linux-gnu/pkg/mod/github.com/spf13/cobra@v0.0.0-20150809222549-c55cdf33856a/command.go:477
github.com/spf13/cobra.(*Command).Execute
	/tmp/docker_run/src/git-lfs/amd64/obj-x86_64-linux-gnu/pkg/mod/github.com/spf13/cobra@v0.0.0-20150809222549-c55cdf33856a/command.go:551
github.com/git-lfs/git-lfs/commands.Run
	/tmp/docker_run/src/git-lfs/amd64/commands/run.go:77
main.main
	/tmp/docker_run/src/git-lfs/amd64/git-lfs.go:35
runtime.main
	/usr/local/go/src/runtime/proc.go:201
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1333
Smudge error
github.com/git-lfs/git-lfs/errors.newWrappedError
	/tmp/docker_run/src/git-lfs/amd64/errors/types.go:184
github.com/git-lfs/git-lfs/errors.NewSmudgeError
	/tmp/docker_run/src/git-lfs/amd64/errors/types.go:270
github.com/git-lfs/git-lfs/lfs.(*GitFilter).Smudge
	/tmp/docker_run/src/git-lfs/amd64/lfs/gitfilter_smudge.go:80
github.com/git-lfs/git-lfs/commands.smudge
	/tmp/docker_run/src/git-lfs/amd64/commands/command_smudge.go:127
github.com/git-lfs/git-lfs/commands.smudgeCommand
	/tmp/docker_run/src/git-lfs/amd64/commands/command_smudge.go:161
github.com/spf13/cobra.(*Command).execute
	/tmp/docker_run/src/git-lfs/amd64/obj-x86_64-linux-gnu/pkg/mod/github.com/spf13/cobra@v0.0.0-20150809222549-c55cdf33856a/command.go:477
github.com/spf13/cobra.(*Command).Execute
	/tmp/docker_run/src/git-lfs/amd64/obj-x86_64-linux-gnu/pkg/mod/github.com/spf13/cobra@v0.0.0-20150809222549-c55cdf33856a/command.go:551
github.com/git-lfs/git-lfs/commands.Run
	/tmp/docker_run/src/git-lfs/amd64/commands/run.go:77
main.main
	/tmp/docker_run/src/git-lfs/amd64/git-lfs.go:35
runtime.main
	/usr/local/go/src/runtime/proc.go:201
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1333

Current time in UTC: 
2018-12-21 23:34:40

ENV:
LocalWorkingDir=/home/halfak/projects/eq_gerrit
LocalGitDir=/home/halfak/projects/eq_gerrit/.git
LocalGitStorageDir=/home/halfak/projects/eq_gerrit/.git
LocalMediaDir=/home/halfak/projects/eq_gerrit/.git/lfs/objects
LocalReferenceDirs=
TempDir=/home/halfak/projects/eq_gerrit/.git/lfs/tmp
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/home/halfak/projects/eq_gerrit/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic
UploadTransfers=basic
GIT_DIR=.git

Client IP addresses:
192.168.1.74 fe80::ce1c:c656:4696:d171

Note that there was a different error caused by a history rewrite in GitHub. I force-pushed a new master to editquality which is built on the old history matching what we've been deploying, with the recent few PRs cherry-picked to catch up with the corrupt master.

It's worth checking out and giving this another try.

I actually saw this before and there is no easy way to properly fix it but there is a very easy way to go around it. Just add gerrit as another remote to your github editquality repo (git remote add gerrit ssh://path/to/gerrit) then simply do git lfs push. It uploads the lfs files on gerrit.

I pushed all of the updates to gerrit. I found out that drafttopic and draftquality are not actually using git lfs. I need to fix that later.

@Ladsgroup I'm pretty sure you have a locally rewritten history for your editquality repo—please pull master from github and delete your rewritten branch. Don't force push until this is fixed, please!

Hey folks. What's the status of this? Can I trust the gerrit repos yet?

Halfak claimed this task.
Halfak triaged this task as Medium priority.
Halfak moved this task from Unorganized to Maintenance/cleanup on the Machine-Learning-Team board.

It looks like @Ladsgroup's hack has resolved this. So I'm closing. Feel free to re-open.

Looks like this work-around isn't working now.

$ git lfs push gerrit master
Locking support detected on remote "gerrit". Consider enabling it with:
  $ git config lfs.https://gerrit.wikimedia.org/r/scoring/ores/articlequality.git/info/lfs.locksverify true
LFS upload failed:cts:  99% (86/87), 2.3 GB | 0 B/s                                                                
  (missing) models/svwiki.wp10.gradient_boosting.model (7b5953f6f237556eed63ff4b3198659245db1bd8e416532e560e286e1b7762b2)
Uploading LFS objects:  99% (86/87), 2.3 GB | 0 B/s, done
hint: Your push was rejected due to missing or corrupt local objects.
hint: You can disable this check with: 'git config lfs.allowincompletepush true'

Nevermind. I needed to do git lfs fetch --all first.