Change Details

Just now we had an outage of gitblit on antimony. Symptom was a very slow response from https://git.wikimedia.org/, and then "Internal error". Solution was to restart gitblit (I didn't find any logs, so root cause is unknown), which took about 2 minutes to start serving requests. During that time we got a different error message from misc-varnish. Thanks to @mmodell and @hoo for reporting & advice. Task: define an Icinga monitor to detect this condition.

Just now we had an outage of gitblit on antimony. Symptom was a very slow response from https://git.wikimedia.org/, and then "Internal error". Solution was to restart gitblit (I didn't find any logs, so root cause is unknown), which took about 2 minutes to start serving requests. During that time we got a different error message from misc-varnish. Thanks to @mmodell and @hoo for reporting & advice. Task: improve existing Icinga monitor to detect this condition.

Just now we had an outage of gitblit on antimony. Symptom was a very slow response from https://git.wikimedia.org/, and then "Internal error". Solution was to restart gitblit (I didn't find any logs, so root cause is unknown), which took about 2 minutes to start serving requests. During that time we got a different error message from misc-varnish. Thanks to @mmodell and @hoo for reporting & advice. Task: define animprove existing Icinga monitor to detect this condition.