Page MenuHomePhabricator
Create Task
Maniphest T212544

Gerrit repo scoring/ores/editquality LFS broken (smudge filter lfs failed)
Closed, ResolvedPublic
Actions

Description

$ git clone https://gerrit.wikimedia.org/r/scoring/ores/editquality eq_gerrit
Cloning into 'eq_gerrit'...
remote: Counting objects: 3778, done
remote: Finding sources: 100% (4661/4661)
remote: Getting sizes: 100% (1216/1216)
remote: Compressing objects: 100% (12246/12246)
remote: Total 4661 (delta 2456), reused 4654 (delta 2452)
Receiving objects: 100% (4661/4661), 3.87 MiB | 0 bytes/s, done.
Resolving deltas: 100% (2456/2456), done.
Checking connectivity... done.
Downloading models/arwiki.damaging.gradient_boosting.model (360 KB)
Error downloading object: models/arwiki.damaging.gradient_boosting.model (7b868ea): Smudge error: Error downloading models/arwiki.damaging.gradient_boosting.model (7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a): [7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a] Object '7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a' not found: [404] Object '7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a' not found

Errors logged to /home/halfak/projects/eq_gerrit/.git/lfs/logs/20181221T173440.706423731.log
Use `git lfs logs last` to view the log.
error: external filter git-lfs smudge -- %f failed 2
error: external filter git-lfs smudge -- %f failed
fatal: models/arwiki.damaging.gradient_boosting.model: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

Related Objects

Event Timeline

Halfak created this task.Dec 21 2018, 11:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 21 2018, 11:36 PM
Halfak added a comment.Dec 21 2018, 11:38 PM

Here's the error log:

git-lfs/2.6.0 (GitHub; linux amd64; go 1.11.1)
git version 2.7.4

$ git-lfs smudge -- models/arwiki.damaging.gradient_boosting.model
Error downloading object: models/arwiki.damaging.gradient_boosting.model (7b868ea): Smudge error: Error downloading models/arwiki.damaging.gradient_boosting.model (7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a): [7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a] Object '7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a' not found: [404] Object '7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a' not found

[404] Object '7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a' not found
[7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a] Object '7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a' not found
github.com/git-lfs/git-lfs/errors.newWrappedError
	/tmp/docker_run/src/git-lfs/amd64/errors/types.go:184
github.com/git-lfs/git-lfs/errors.Wrapf
	/tmp/docker_run/src/git-lfs/amd64/errors/errors.go:85
github.com/git-lfs/git-lfs/tq.(*TransferQueue).enqueueAndCollectRetriesFor
	/tmp/docker_run/src/git-lfs/amd64/tq/transfer_queue.go:470
github.com/git-lfs/git-lfs/tq.(*TransferQueue).collectBatches.func1
	/tmp/docker_run/src/git-lfs/amd64/tq/transfer_queue.go:361
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1333
Error downloading models/arwiki.damaging.gradient_boosting.model (7b868ea3fbdc7572e940dad040dbbf8e7d3609cc14cefb782379151aed14d07a)
github.com/git-lfs/git-lfs/errors.newWrappedError
	/tmp/docker_run/src/git-lfs/amd64/errors/types.go:184
github.com/git-lfs/git-lfs/errors.Wrapf
	/tmp/docker_run/src/git-lfs/amd64/errors/errors.go:85
github.com/git-lfs/git-lfs/lfs.(*GitFilter).downloadFile
	/tmp/docker_run/src/git-lfs/amd64/lfs/gitfilter_smudge.go:110
github.com/git-lfs/git-lfs/lfs.(*GitFilter).Smudge
	/tmp/docker_run/src/git-lfs/amd64/lfs/gitfilter_smudge.go:71
github.com/git-lfs/git-lfs/commands.smudge
	/tmp/docker_run/src/git-lfs/amd64/commands/command_smudge.go:127
github.com/git-lfs/git-lfs/commands.smudgeCommand
	/tmp/docker_run/src/git-lfs/amd64/commands/command_smudge.go:161
github.com/spf13/cobra.(*Command).execute
	/tmp/docker_run/src/git-lfs/amd64/obj-x86_64-linux-gnu/pkg/mod/github.com/spf13/cobra@v0.0.0-20150809222549-c55cdf33856a/command.go:477
github.com/spf13/cobra.(*Command).Execute
	/tmp/docker_run/src/git-lfs/amd64/obj-x86_64-linux-gnu/pkg/mod/github.com/spf13/cobra@v0.0.0-20150809222549-c55cdf33856a/command.go:551
github.com/git-lfs/git-lfs/commands.Run
	/tmp/docker_run/src/git-lfs/amd64/commands/run.go:77
main.main
	/tmp/docker_run/src/git-lfs/amd64/git-lfs.go:35
runtime.main
	/usr/local/go/src/runtime/proc.go:201
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1333
Smudge error
github.com/git-lfs/git-lfs/errors.newWrappedError
	/tmp/docker_run/src/git-lfs/amd64/errors/types.go:184
github.com/git-lfs/git-lfs/errors.NewSmudgeError
	/tmp/docker_run/src/git-lfs/amd64/errors/types.go:270
github.com/git-lfs/git-lfs/lfs.(*GitFilter).Smudge
	/tmp/docker_run/src/git-lfs/amd64/lfs/gitfilter_smudge.go:80
github.com/git-lfs/git-lfs/commands.smudge
	/tmp/docker_run/src/git-lfs/amd64/commands/command_smudge.go:127
github.com/git-lfs/git-lfs/commands.smudgeCommand
	/tmp/docker_run/src/git-lfs/amd64/commands/command_smudge.go:161
github.com/spf13/cobra.(*Command).execute
	/tmp/docker_run/src/git-lfs/amd64/obj-x86_64-linux-gnu/pkg/mod/github.com/spf13/cobra@v0.0.0-20150809222549-c55cdf33856a/command.go:477
github.com/spf13/cobra.(*Command).Execute
	/tmp/docker_run/src/git-lfs/amd64/obj-x86_64-linux-gnu/pkg/mod/github.com/spf13/cobra@v0.0.0-20150809222549-c55cdf33856a/command.go:551
github.com/git-lfs/git-lfs/commands.Run
	/tmp/docker_run/src/git-lfs/amd64/commands/run.go:77
main.main
	/tmp/docker_run/src/git-lfs/amd64/git-lfs.go:35
runtime.main
	/usr/local/go/src/runtime/proc.go:201
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1333

Current time in UTC: 
2018-12-21 23:34:40

ENV:
LocalWorkingDir=/home/halfak/projects/eq_gerrit
LocalGitDir=/home/halfak/projects/eq_gerrit/.git
LocalGitStorageDir=/home/halfak/projects/eq_gerrit/.git
LocalMediaDir=/home/halfak/projects/eq_gerrit/.git/lfs/objects
LocalReferenceDirs=
TempDir=/home/halfak/projects/eq_gerrit/.git/lfs/tmp
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/home/halfak/projects/eq_gerrit/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic
UploadTransfers=basic
GIT_DIR=.git

Client IP addresses:
192.168.1.74 fe80::ce1c:c656:4696:d171
Kizule added a subscriber: Kizule.Dec 22 2018, 12:56 AM

Maybe this: https://github.com/git-lfs/git-lfs/issues/911

awight added a subscriber: awight.Dec 22 2018, 1:10 AM

Note that there was a different error caused by a history rewrite in GitHub. I force-pushed a new master to editquality which is built on the old history matching what we've been deploying, with the recent few PRs cherry-picked to catch up with the corrupt master.

It's worth checking out and giving this another try.

hashar removed a project: Gerrit.Dec 22 2018, 7:34 AM
Ladsgroup added a subscriber: Ladsgroup.Dec 22 2018, 3:05 PM

I actually saw this before and there is no easy way to properly fix it but there is a very easy way to go around it. Just add gerrit as another remote to your github editquality repo (git remote add gerrit ssh://path/to/gerrit) then simply do git lfs push. It uploads the lfs files on gerrit.

Ladsgroup added a comment.Dec 22 2018, 4:24 PM

I pushed all of the updates to gerrit. I found out that drafttopic and draftquality are not actually using git lfs. I need to fix that later.

awight added a comment.Jan 2 2019, 5:42 PM

@Ladsgroup I'm pretty sure you have a locally rewritten history for your editquality repo—please pull master from github and delete your rewritten branch. Don't force push until this is fixed, please!

awight mentioned this in T212530: Rebuild models for new revscoring (2.3.0).Jan 2 2019, 11:28 PM
Halfak added a comment.Jan 3 2019, 3:18 PM

Hey folks. What's the status of this? Can I trust the gerrit repos yet?

Ladsgroup mentioned this in T212818: LFS objects are not mirroring from Github through Phab to Gerrit consistently.Jan 9 2019, 10:08 AM

I don't think this is a big issue: T212818#4865068

Halfak closed this task as Resolved.Feb 5 2019, 10:49 PM
Halfak claimed this task.
Halfak triaged this task as Medium priority.
Halfak moved this task from Unorganized to Maintenance/cleanup on the Machine-Learning-Team board.

It looks like @Ladsgroup's hack has resolved this. So I'm closing. Feel free to re-open.

Halfak reopened this task as Open.May 10 2019, 1:55 PM

Looks like this work-around isn't working now.

$ git lfs push gerrit master
Locking support detected on remote "gerrit". Consider enabling it with:
  $ git config lfs.https://gerrit.wikimedia.org/r/scoring/ores/articlequality.git/info/lfs.locksverify true
LFS upload failed:cts:  99% (86/87), 2.3 GB | 0 B/s                                                                
  (missing) models/svwiki.wp10.gradient_boosting.model (7b5953f6f237556eed63ff4b3198659245db1bd8e416532e560e286e1b7762b2)
Uploading LFS objects:  99% (86/87), 2.3 GB | 0 B/s, done
hint: Your push was rejected due to missing or corrupt local objects.
hint: You can disable this check with: 'git config lfs.allowincompletepush true'
Halfak added a comment.May 10 2019, 2:22 PM

Nevermind. I needed to do git lfs fetch --all first.

Halfak closed this task as Resolved.May 10 2019, 2:23 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL