Page MenuHomePhabricator

Unable to clone Mediawiki core repo due to timeout
Closed, ResolvedPublic

Description

There has been a strange error that I've been encountering while trying to clone any repo via gerrit.

Steps to reproduce error with sandbox:

git clone ssh://alqaholic007@gerrit.wikimedia.org:29418/sandbox

This gets stuck at 93% and exits with the following:

fatal: internal server error
fatal: the remote end hung up upon initial contact

Steps to reproduce error with mediawiki/core:

git clone ssh://alqaholic007@gerrit.wikimedia.org:29418/mediawiki/core.git mediawiki

This exists with:

fatal: internal server error
fatal: the remote end hung up upon initial contact

without cloning any file.

Initially I thought this might be a network error. But Zulip, Phabricator and everything else seems to be loading fine.

OS Version: OSX Mojave
Git Version: 2.23.0

Event Timeline

@AlQaholic007 Have you added the ssh key to Gerrit ??

Hi,

You are hitting a some kind of timeout on Gerrit side. All times below are UTC:

[2020-04-04 10:50:15,118 +0000] 7b631a45 alqaholic007 a/8103 LOGIN FROM x.x.x.x
[2020-04-04 10:52:27,466 +0000] 7b631a45 alqaholic007 a/8103 git-upload-pack./mediawiki/core.git 2ms 125785ms 128

Aborted due to:

[2020-04-04 10:52:27,465] [SSH git-upload-pack /mediawiki/core.git (alqaholic007)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user alqaholic007) during git-upload-pack '/mediawiki/core.git
+-- 15 lines: at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:792)-----------------------------------------------------------------------------------------------------------Caused by: java.net.SocketTimeoutException: waitForCondition(Window[server/remote](ChannelSession[id=0, recipient=0]-ServerSessionImpl[alqaholic007@/x.x.x.x:yy])) timeout exceeded: 30000
[2020-04-04 11:19:23,100 +0000] 63b2a876 alqaholic007 a/8103 LOGIN FROM x.x.x.x
[2020-04-04 11:20:02,867 +0000] 63b2a876 alqaholic007 a/8103 git-upload-pack./mediawiki/core.git 2ms 34032ms 128
[2020-04-04 11:26:31,131 +0000] 63b2a876 alqaholic007 a/8103 LOGOUT
[2020-04-04 11:20:02,866] [SSH git-upload-pack /mediawiki/core.git (alqaholic007)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user alqaholic007) during git-upload-pack '/mediawiki/core.git'
org.eclipse.jgit.transport.UploadPackInternalServerErrorException
Caused by: java.net.SocketTimeoutException: waitForCondition(Window[server/remote](ChannelSession[id=0, recipient=0]-ServerSessionImpl[alqaholic007@/x.x.x.x:yy])) timeout exc```eeded: 30000

The whole stack trace:

Caused by: java.net.SocketTimeoutException: waitForCondition(Window[server/remote](ChannelSession[id=0, recipient=0]-ServerSessionImpl[alqaholic007@/x.x.x.x:yyyy])) timeout exceeded: 30000
        at org.apache.sshd.common.channel.Window.waitForCondition(Window.java:304)
        at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:251)
        at org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:179)
        at org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:120)
        at org.eclipse.jgit.transport.UploadPack$ResponseBufferedOutputStream.write(UploadPack.java:1610)
        at org.eclipse.jgit.transport.PacketLineOut.writePacket(PacketLineOut.java:142)
        at org.eclipse.jgit.transport.RefAdvertiser$PacketLineOutRefAdvertiser.advertiseId(RefAdvertiser.java:111)
        at org.eclipse.jgit.transport.RefAdvertiser.advertiseAny(RefAdvertiser.java:330)
        at org.eclipse.jgit.transport.RefAdvertiser.send(RefAdvertiser.java:274)
        at org.eclipse.jgit.transport.UploadPack.sendAdvertisedRefs(UploadPack.java:926)
        at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:731)
        ... 14 more

I think you are hitting this timeout:

[[sshd.waitTimeout]]sshd.waitTimeout::

Time in seconds after which the server automatically terminates
connections waiting for a server operation to complete, like for instance
cloning a very large repo with lots of refs.
...
By default, 30s.

To unlock you, you can try to clone a single branch with --single-branch or just skip cloning the history entirely with --depth 1 (which implies --single-branch).

git clone --depth 1 ssh://alqaholic007@gerrit.wikimedia.org:29418/sandbox

The history can later be fetched with git fetch --unshallow but I guess that will timeout.

Aklapper renamed this task from Unable to clone Mediawiki core repo to Unable to clone Mediawiki core repo due to timeout.Apr 4 2020, 12:04 PM
This comment was removed by Soham.
This comment was removed by Soham.

@AlQaholic007: Why don/t you just try, but ask instead?

This comment was removed by Soham.

@AlQaholic007: Why don/t you just try, but ask instead?

Just did what @hashar suggested. I was able to clone the mediawiki core master branch. However I am unable to run:
git review -d <patch number> because of the same error

Hmm. I guess the fall back is to clone from the GitHub mirror.

In the repository you already cloned, you can fetch objects from Github using:

git fetch https://github.com/wikimedia/mediawiki/

This way you will get all the objects and that might work around the timeout.

Was this resolved, or is it still a problem?

Hi @zeljkofilipin this is still unresolved. I am not too sure if it's a network issue but I was able to work around temporarily with the alternative suggested by @hashar.

Can you try again with some debug logs turned on?

GIT_TRACE=1 git clone ssh://alqaholic007@gerrit.wikimedia.org:29418/sandbox ; date

And over https, which should not be affected by the Gerrit ssh timeout I have mentioned earlier:

GIT_TRACE=1 git clone https://gerrit.wikimedia.org/r/sandbox ; date

This issue has been resolved now. I think it was related to network issues. Thank you everyone for help ๐Ÿ˜