Page MenuHomePhabricator

Git.wikimedia.org keeps going down
Closed, DuplicatePublic

Description

Request: GET http://git.wikimedia.org/, from *.*.*.* via cp1043 cp1043 ([208.80.154.241]:80), Varnish XID 1210028928
Forwarded for: *.*.*.*
Error: 503, Service Unavailable at Sun, 12 Oct 2014 10:16:15 GMT


Version: wmf-deployment
Severity: major
URL: http://git.wikimedia.org/

Details

Reference
bz71974

Event Timeline

bzimport raised the priority of this task from to Unbreak Now!.Nov 22 2014, 3:49 AM
bzimport added projects: Gerrit, acl*sre-team.
bzimport set Reference to bz71974.
bzimport added a subscriber: Unknown Object (MLST).

Up and down could be seen this weekend on #wikimedia-operations:

<icinga-wm> PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds
<icinga-wm> RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 54211 bytes in 4.834 second response time

According to <_joe_> this was due to gitblit being unstable and <paravoid> restarted java which made this problem disappear.
In general, gitblit has stability issues.

Dzahn renamed this task from Git.wikimedia.org is down to Git.wikimedia.org keeps going down.Feb 3 2015, 11:19 PM
Dzahn reopened this task as Open.
Dzahn lowered the priority of this task from Unbreak Now! to Medium.
Dzahn set Security to None.
Dzahn added a project: Gitblit-Deprecate.

echo "Gitblit has been restarted $(wget -O - https://wikitech.wikimedia.org/wiki/Server_Admin_Log 2> /dev/null| grep "restarted gitblit" | wc -l) times (plus when we did not log)"

Gitblit has been restarted 7 times (plus when we did not log)

http://git.wikimedia.org/ is giving me a lightly-styled Apache HTML error page
Internal error
Return to home page

It's HTTP status 500 (see below), not status 503 like task description.

HTTP/1.1 500 Server Error
    use_proxy = "no";
Content-Type: text/html;charset=UTF-8
Content-Language: en-US
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Server: Jetty(7.6.8.v20121106)
Content-Length: 1406
Accept-Ranges: bytes
Date: Thu, 26 Mar 2015 22:30:33 GMT
X-Varnish: 983879703
Age: 0
Via: 1.1 varnish
Connection: keep-alive
X-Cache: cp1043 miss (0)

Would T51371: GitBlit main page sets Cache-Control: no-cache help?

Not much. Getting rid of the blasted tool would help more (T752)

@Melos please see progress on T83702 , added you there too so we can close this one as a duplicate

Dzahn claimed this task.

@Dzahn, Isn't this the correct way to merge dups?