Page MenuHomePhabricator

Diffusion repository can't be cloned: 500 errors (research-ores-editquality)
Closed, ResolvedPublic

Description

$ git clone https://phabricator.wikimedia.org/diffusion/1913/research-ores-editquality.git
Cloning into 'research-ores-editquality'...
error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504 Gateway Time-out
fatal: The remote end hung up unexpectedly

This is likely because the repository has become very large. We (Scoring-platform-team) are looking into git-fat for moving off some off the large files, but in the meantime, we need to work with the repository.

Event Timeline

Halfak created this task.Feb 3 2017, 4:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 3 2017, 4:23 PM
demon added a subscriber: demon.Feb 3 2017, 4:35 PM

To be fair, removing the files still keep a large history as the old blobs will still be stored in your objects. But it will keep the size from growing, so not a bad goal at all :D

Paladox added a subscriber: Paladox.Feb 3 2017, 4:35 PM
Halfak added a comment.Feb 3 2017, 4:37 PM

Right. We'd re-write the history in order to manage that.

Halfak added a comment.Feb 3 2017, 4:39 PM

For the curious, you can do this with a git filter-branch

See https://github.com/git-lfs/git-lfs/issues/326

Halfak added a comment.Feb 3 2017, 4:57 PM

One more note. This is "High" priority for us because it's blocked all deployments for ORES -- a production-level service. I encourage Repository-Admins to consider prioritizing this "High" as well.

Paladox added a subscriber: mmodell.Feb 3 2017, 5:29 PM

We could raise http://httpd.apache.org/docs/2.0/mod/core.html#timeout on apache on phabricator if that's ok with @mmodell ?

demon added a comment.Feb 3 2017, 5:29 PM

No, there's better solutions here than just raising limits. Slow things should be fast, not allowed to be slow.

try cloning over ssh instead of https. Phabricator has a known limitation with https (due to the request being processed and buffered through php-land)

hashar added a subscriber: hashar.Feb 3 2017, 8:59 PM

try passing to git clone --depth 1 to reduce the history to the last commit. Or alternatively only get the branch you are interested in with --single-branch (which is implied by --depth).

Then it still does a bunch of work on the server side, takes me a 1:20min to download it:

$ time git clone --depth 1 --recurse --shallow-submodules https://phabricator.wikimedia.org/diffusion/1913/research-ores-editquality.git
Cloning into 'research-ores-editquality'...
remote: Counting objects: 163, done.
remote: Compressing objects: 100% (145/145), done.
remote: Total 163 (delta 68), reused 28 (delta 14)
Receiving objects: 100% (163/163), 131.05 MiB | 11.46 MiB/s, done.
Resolving deltas: 100% (68/68), done.

real    1m20.448s
user    0m6.992s
sys     0m1.168s

Running git clone with strace shows that the client is just doing a poll() over the socket, waiting for the server to reply. So I am assuming most of the work is being done on the server side that tries to figure out which objects to send.

Note also the Diffusion repository has references for all GitHub pull requests (refs/pull/*). Though git clone should not grab them.

I don't know if this is the bug you are hitting but this might be related to an upstream bug: https://secure.phabricator.com/T4369

It fails when doing the scap deploy so I'm not sure if there is a way to bypass git commands using --shallow. I tried ssh in beta tin, it failed because no ssh key was defined.

What about annon git:// ? You won't be able to push things for code review through that but you will be able to clone through it.

@mmodell or @demon do you think using git:// would be a good idea here. Because using ssh requires your key but annon git:// dosent.

we will probably want to setup git://. Because unlike gerrit, http works there better but in phab it seems to have some limitations.

Which makes it a good idea to setup the anon git://. Otherwise users that run git submodule update will fail unless they are a user in phab and have a ssh key attach to there account.

demon added a comment.Feb 6 2017, 2:10 AM

we will probably want to setup git://. Because unlike gerrit, http works there better but in phab it seems to have some limitations.
Which makes it a good idea to setup the anon git://. Otherwise users that run git submodule update will fail unless they are a user in phab and have a ssh key attach to there account.

For the 800th time, we're not going to use the anonymous git:// protocol. It's insecure and offers no performance benefits.

Halfak added a comment.Feb 7 2017, 7:15 PM

Confirmed that this all now works with ssh-git. I'm resolving. Thanks for your help!

Halfak closed this task as Resolved.Feb 7 2017, 7:15 PM
Restricted Application added a project: artificial-intelligence. · View Herald TranscriptJul 3 2017, 5:50 PM