Page MenuHomePhabricator

gitblit (git.wikimedia.org) times out: 503, Service Unavailable
Closed, ResolvedPublic

Description

git.wikimedia.org just times out for me.

Request: GET http://git.wikimedia.org/, from 91.153.135.115 via cp1043 cp1043 ([208.80.154.241]:80), Varnish XID 1001503963
Forwarded for: 91.153.135.115
Error: 503, Service Unavailable at Tue, 14 Apr 2015 08:44:45 GMT

Cf. T73974: Git.wikimedia.org keeps going down

Event Timeline

Nikerabbit raised the priority of this task from to Needs Triage.
Nikerabbit updated the task description. (Show Details)
Nikerabbit subscribed.
Nemo_bis added a project: Gerrit.
Nemo_bis set Security to None.
Nemo_bis subscribed.

Works for me in the browser, but took a while to load.

It's back down. Probably need to ping someone on IRC who can restart it - I think that usually fixes it.

Gitblit is notoriously unstable. Someone told me a while ago that it tends to run out of memory (or max out the CPU, I forget which) when old Gerrit branches are being cleaned up. The intention is to eventually get rid of gitblit and just use Phabricator's repository browser.

I asked on #wikimedia-operations where icinga-wm also brought up:

PROBLEM - git.wikimedia.org on antimony is CRITICAL - Socket timeout after 10 seconds

And _joe_ mentioned that he had already restarted it this (European) morning. But down again it seems.

Aklapper renamed this task from Unable to access git.wikimedia.org to Unable to access gitblit (git.wikimedia.org).Apr 14 2015, 12:38 PM
Aklapper renamed this task from Unable to access gitblit (git.wikimedia.org) to gitblit (git.wikimedia.org) times out: 503, Service Unavailable.Apr 14 2015, 12:45 PM
Aklapper added subscribers: hashar, greg.

gitblit repeatedly goes down after restarts.
Issue needs further investigation (and _joe_ didn't have time to dive deeper).

Aklapper claimed this task.

Currently seems to work for me again...

Ciencia_Al_Poder subscribed.

Not sure if reopening this is the right thing to do, but gitblit is down again...

Request: GET http://git.wikimedia.org/, from 83.45.71.1 via cp1044 cp1044 ([208.80.154.241]:80), Varnish XID 275036852
Forwarded for: 83.45.71.1
Error: 503, Service Unavailable at Wed, 17 Jun 2015 20:02:02 GMT

Ciencia_Al_Poder claimed this task.

It works now, although very very slow

is down again... well, since yesterday IIRC