Page MenuHomePhabricator

Fetching from Gerrit is slow or broken
Closed, ResolvedPublic

Description

Fetching from Gerrit seems to be very slow right now, and sometimes it breaks completely. A few times I had to wait for several minutes (up to 5) before the fetch went through; once, I got an error instead:

$ git fetch
fatal: internal server error
fatal: protocol error: bad line length character: tern

Experienced in mediawiki/core.git and mediawiki/extensions/Wikibase.git so far.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 3 2019, 12:40 PM
greg added a subscriber: greg.Jun 3 2019, 2:52 PM
05:44:17 <+hashar> the long story is https://phabricator.wikimedia.org/T103990
05:44:34 <+hashar> and the fix is to switch to git protocol v2 which our Gerrit does not support yet (gotta upgrade it)
05:45:58 <+hashar> and maybe the path from wikimedia office to gerrit has some congestion at some point. You can try  traceroute gerrit.wikimedia.org

Probably a duplicate of that mentioned bug or a network issue with the WMDE office <-> gerrit path.

Lucas_Werkmeister_WMDE closed this task as Resolved.Jun 3 2019, 4:10 PM

T103990: Gerrit upload-pack send ALL references causing massive network I/O on common operations is definitely part of this, yes, and it looks like there was some other network congestion on a part of the path from the WMDE office to Gerrit – forcing an SSH ProxyJump through bast3002.wikimedia.org (esams) was an effective workaround for me. And by now fetches are fast again without that jump too, so I guess this is resolved for now.

That was definitely a network congestion between Gerrit and the internet. I don't have much details though.

Using bast3002 is definitely a nice hack.