To be fair, removing the files still keep a large history as the old blobs will still be stored in your objects. But it will keep the size from growing, so not a bad goal at all :D

Right. We'd re-write the history in order to manage that.

For the curious, you can do this with a git filter-branch

See https://github.com/git-lfs/git-lfs/issues/326

One more note. This is "High" priority for us because it's blocked all deployments for ORES -- a production-level service. I encourage Diffusion-Repository-Administrators to consider prioritizing this "High" as well.

@Halfak looking on google, multiple answers are it's the proxy.

See http://stackoverflow.com/questions/18501296/504-gateway-timeout-on-gitlab-http-urls

and

https://www.google.co.uk/#q=error:+RPC+failed%3B+HTTP+504+curl+22+The+requested+URL+returned+error:+504+Gateway+Time-out

but i found this that says something different.

https://answers.atlassian.com/questions/21630795/git-clone-failing

We could raise http://httpd.apache.org/docs/2.0/mod/core.html#timeout on apache on phabricator if that's ok with @mmodell ?

No, there's better solutions here than just raising limits. Slow things should be fast, not allowed to be slow.

try cloning over ssh instead of https. Phabricator has a known limitation with https (due to the request being processed and buffered through php-land)

try passing to git clone --depth 1 to reduce the history to the last commit. Or alternatively only get the branch you are interested in with --single-branch (which is implied by --depth).

Then it still does a bunch of work on the server side, takes me a 1:20min to download it:

$ time git clone --depth 1 --recurse --shallow-submodules https://phabricator.wikimedia.org/diffusion/1913/research-ores-editquality.git
Cloning into 'research-ores-editquality'...
remote: Counting objects: 163, done.
remote: Compressing objects: 100% (145/145), done.
remote: Total 163 (delta 68), reused 28 (delta 14)
Receiving objects: 100% (163/163), 131.05 MiB | 11.46 MiB/s, done.
Resolving deltas: 100% (68/68), done.

real    1m20.448s
user    0m6.992s
sys     0m1.168s

Running git clone with strace shows that the client is just doing a poll() over the socket, waiting for the server to reply. So I am assuming most of the work is being done on the server side that tries to figure out which objects to send.

Note also the Diffusion repository has references for all GitHub pull requests (refs/pull/*). Though git clone should not grab them.

I don't know if this is the bug you are hitting but this might be related to an upstream bug: https://secure.phabricator.com/T4369

It fails when doing the scap deploy so I'm not sure if there is a way to bypass git commands using --shallow. I tried ssh in beta tin, it failed because no ssh key was defined.

What about annon git:// ? You won't be able to push things for code review through that but you will be able to clone through it.

@mmodell or @demon do you think using git:// would be a good idea here. Because using ssh requires your key but annon git:// dosent.

I've switched to git-ssh in https://gerrit.wikimedia.org/r/335998.

we will probably want to setup git://. Because unlike gerrit, http works there better but in phab it seems to have some limitations.

Which makes it a good idea to setup the anon git://. Otherwise users that run git submodule update will fail unless they are a user in phab and have a ssh key attach to there account.

In T157141#2998631, @Paladox wrote:

we will probably want to setup git://. Because unlike gerrit, http works there better but in phab it seems to have some limitations.

Which makes it a good idea to setup the anon git://. Otherwise users that run git submodule update will fail unless they are a user in phab and have a ssh key attach to there account.

For the 800th time, we're not going to use the anonymous git:// protocol. It's insecure and offers no performance benefits.

Confirmed that this all now works with ssh-git. I'm resolving. Thanks for your help!