Page MenuHomePhabricator

gitblit (git.wikimedia.org) times out: 503, Service Unavailable
Closed, ResolvedPublic

Description

git.wikimedia.org just times out for me.

Request: GET http://git.wikimedia.org/, from 91.153.135.115 via cp1043 cp1043 ([208.80.154.241]:80), Varnish XID 1001503963
Forwarded for: 91.153.135.115
Error: 503, Service Unavailable at Tue, 14 Apr 2015 08:44:45 GMT

Cf. T73974: Git.wikimedia.org keeps going down

Event Timeline

Nikerabbit raised the priority of this task from to Needs Triage.
Nikerabbit updated the task description. (Show Details)
Nikerabbit added a subscriber: Nikerabbit.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 14 2015, 8:46 AM
Nemo_bis updated the task description. (Show Details)Apr 14 2015, 9:15 AM
Nemo_bis added a project: Gerrit.
Nemo_bis set Security to None.
Nemo_bis added a subscriber: Nemo_bis.
Aklapper triaged this task as High priority.Apr 14 2015, 9:33 AM

Works for me in the browser, but took a while to load.

TTO added a subscriber: TTO.Apr 14 2015, 11:06 AM

It's back down. Probably need to ping someone on IRC who can restart it - I think that usually fixes it.

Gitblit is notoriously unstable. Someone told me a while ago that it tends to run out of memory (or max out the CPU, I forget which) when old Gerrit branches are being cleaned up. The intention is to eventually get rid of gitblit and just use Phabricator's repository browser.

I asked on #wikimedia-operations where icinga-wm also brought up:

PROBLEM - git.wikimedia.org on antimony is CRITICAL - Socket timeout after 10 seconds

And _joe_ mentioned that he had already restarted it this (European) morning. But down again it seems.

Aklapper renamed this task from Unable to access git.wikimedia.org to Unable to access gitblit (git.wikimedia.org).Apr 14 2015, 12:38 PM
Aklapper renamed this task from Unable to access gitblit (git.wikimedia.org) to gitblit (git.wikimedia.org) times out: 503, Service Unavailable.Apr 14 2015, 12:45 PM
Aklapper added subscribers: hashar, greg.

gitblit repeatedly goes down after restarts.
Issue needs further investigation (and _joe_ didn't have time to dive deeper).

hashar removed a subscriber: hashar.Apr 14 2015, 1:59 PM
Aklapper closed this task as Resolved.Apr 22 2015, 10:10 PM
Aklapper claimed this task.

Currently seems to work for me again...

Ciencia_Al_Poder reopened this task as Open.Jun 17 2015, 8:08 PM
Ciencia_Al_Poder added a subscriber: Ciencia_Al_Poder.

Not sure if reopening this is the right thing to do, but gitblit is down again...

Request: GET http://git.wikimedia.org/, from 83.45.71.1 via cp1044 cp1044 ([208.80.154.241]:80), Varnish XID 275036852
Forwarded for: 83.45.71.1
Error: 503, Service Unavailable at Wed, 17 Jun 2015 20:02:02 GMT

Aklapper removed Aklapper as the assignee of this task.Jun 18 2015, 8:15 AM
Ciencia_Al_Poder closed this task as Resolved.Jun 18 2015, 9:14 AM
Ciencia_Al_Poder claimed this task.

It works now, although very very slow

Ciencia_Al_Poder removed Ciencia_Al_Poder as the assignee of this task.Jun 18 2015, 9:14 AM
Ciencia_Al_Poder updated the task description. (Show Details)
Ciencia_Al_Poder reopened this task as Open.Jul 2 2015, 2:17 PM

is down again... well, since yesterday IIRC

Ciencia_Al_Poder closed this task as Resolved.Jul 2 2015, 3:05 PM
Ciencia_Al_Poder assigned this task to Dzahn.

resolved for now per T104356#1421113