Page MenuHomePhabricator

Create gerrit mirrors for all github-based ORES repos
Closed, ResolvedPublic

Description

This task is tracking work already in progress. We're considering a workaround (successfully demonstrated in the beta cluster) where scap won't rewrite our submodule URLs, to avoid an idiosyncracy in git-lfs. Unfortunately, our submodules are hosted on phabricator for historical reasons, but aren't accessible from deployment targets. Therefore, we're creating additional git mirrors in gerrit which will be accessible during deployment.

Event Timeline

awight triaged this task as Normal priority.Apr 12 2018, 12:02 AM
awight created this task.
awight updated the task description. (Show Details)Apr 12 2018, 6:10 PM
mmodell closed this task as Resolved.Apr 12 2018, 8:03 PM
mmodell updated the task description. (Show Details)

Sorry it took me so long. I just saw the update.

Sorry it took me so long. I just saw the update.

<3 I'd say that resolving a task just a few hours after it's defined is setting a new standard for "so long".

@awight: I'd like to unblock your project as much as possible and the week is nearly over so a few hours add up.

Ladsgroup reopened this task as Open.Aug 20 2018, 7:05 PM
Ladsgroup added a subscriber: Ladsgroup.

After migrating the github repos to use git lfs, now it returns this error:

amsa@C235:~/ores-prod-deploy/submodules/articlequality$ git reset --hard origin/master
Downloading models/enwiki.nettrom_wp10.gradient_boosting.model (60 MB)
git pError downloading object: models/enwiki.nettrom_wp10.gradient_boosting.model (9b52295): Smudge error: Error downloading models/enwiki.nettrom_wp10.gradient_boosting.model (9b52295b6bd398ab72f2a6bdace7450672f47d502b893dd8d3b333bfd8077e37): [9b52295b6bd398ab72f2a6bdace7450672f47d502b893dd8d3b333bfd8077e37] Object '9b52295b6bd398ab72f2a6bdace7450672f47d502b893dd8d3b333bfd8077e37' not found: [404] Object '9b52295b6bd398ab72f2a6bdace7450672f47d502b893dd8d3b333bfd8077e37' not found

It's great we've got so far, just one nudge and lfs for all models is in production \o/

@Ladsgroup Please be sure to share the update procedure with us once you're ready. I get this warning:

Your branch and 'origin/master' have diverged,

A "pull" is clearly wrong here…. and should I run the reflog expire and gc prune commands to release space?

Yeah, The thing is the whole history is rewritten now, so the hashes of commits are different and now your system thinks they are diverged. Nothing to worry about, you basically need to throw everything and build it again :D. I think I've got a solution for this :P

awight added a comment.EditedAug 20 2018, 9:29 PM

I get an error on the same object:

$ git status
On branch master
Your branch and 'origin/master' have diverged,
and have 251 and 251 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)

nothing to commit, working tree clean

$ git checkout origin/master
Downloading models/enwiki.nettrom_wp10.gradient_boosting.model (60 MB)
Error downloading object: models/enwiki.nettrom_wp10.gradient_boosting.model (9b52295): Smudge error: Error downloading models/enwiki.nettrom_wp10.gradient_boosting.model (9b52295b6bd398ab72f2a6bdace7450672f47d502b893dd8d3b333bfd8077e37): batch response: Authorization error: https://phabricator.wikimedia.org/source/wikiclass.git/info/lfs/objects/batch
Check that you have proper access to the repository

Errors logged to /X/ores-prod-deploy/.git/modules/submodules/articlequality/lfs/objects/logs/20180820T142630.554616.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: models/enwiki.nettrom_wp10.gradient_boosting.model: smudge filter lfs failed

In the future it seems prudent to do this sort of rewrite into a named branch, but I guess that's not part of the LFS migration recommendations.

Here's the error log mentioned above,

In the future it seems prudent to do this sort of rewrite into a named branch, but I guess that's not part of the LFS migration recommendations.

I have two backups just in case T197097#4515707

FYI, completely removing and adding a submodule didn't work either.

In the future it seems prudent to do this sort of rewrite into a named branch, but I guess that's not part of the LFS migration recommendations.

I have two backups just in case T197097#4515707

Thank you for that! I'm thinking of something else though, that you could do this entire migration as a dry run, and only rename over to the master branch once we've proven that things like continuous integration still work. Luckily, I think that for this repo it's fine because we never need it "bare", it's only a submodule in production so we can simply skip over any corrupted commits.

It seems git-lfs does not have any support for mirroring. I think that we will have to roll our own solution for mirroring from github to gerrit.

The phabricator repos themselves are broken:

amsa@C235:~$ git clone http://phabricator.wikimedia.org/source/editquality.git editquality-phab
Cloning into 'editquality-phab'...
warning: redirecting to https://phabricator.wikimedia.org/source/editquality.git/
remote: Counting objects: 3744, done.
remote: Compressing objects: 100% (2644/2644), done.
remote: Total 3744 (delta 1909), reused 2716 (delta 1054)
Receiving objects: 100% (3744/3744), 3.77 MiB | 6.34 MiB/s, done.
Resolving deltas: 100% (1909/1909), done.
Downloading models/arwiki.damaging.gradient_boosting.model (378 KB)
Error downloading object: models/arwiki.damaging.gradient_boosting.model (c1f9cb0): Smudge error: Error downloading models/arwiki.damaging.gradient_boosting.model (c1f9cb02bac12da3a421ee146c34a9fe7fa89a4993cd471d2f9bbd0ce977719d): batch response: Authorization error: http://phabricator.wikimedia.org/source/editquality.git/info/lfs/objects/batch
Check that you have proper access to the repository

Errors logged to /home/amsa/editquality-phab/.git/lfs/logs/20180906T173459.087885996.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: models/arwiki.damaging.gradient_boosting.model: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

Phabricator doesn't have proper git-lfs support and I've been told not to put any resources into phabricator's git hosting infrastructure. I want to express my exasperation at this situation. I don't like it really at all. Unfortunately it simply isn't a priority and gerrit is apparently our only officially supported git service for now.

awight removed a subscriber: awight.Mar 21 2019, 4:03 PM
mmodell closed this task as Resolved.Mar 25 2019, 5:47 PM