Page MenuHomePhabricator

gerrit.wikimedia.org is down
Closed, ResolvedPublic

Assigned To
None
Authored By
Daimona
Mar 16 2019, 10:13 AM
Referenced Files
None
Tokens
"Cup of Joe" token, awarded by Mvolz."The World Burns" token, awarded by MGChecker."The World Burns" token, awarded by Base."Like" token, awarded by Ammarpad."The World Burns" token, awarded by matej_suchanek.

Description

What it says on the tin. Trying to open any page results in a connection refused error. Already checked that it's not only me, and icinga has a few errors about that from 10 minutes ago.

Event Timeline

Daimona triaged this task as Unbreak Now! priority.Mar 16 2019, 10:14 AM
<chasemp> !log stop apache on cobalt for maintenance

Well if gerrit is on maintenance and apache has been disconnected then this downtime is to be expected :-)

Given that this is not an involuntary outage, shall we degrade from UBN?

@MarcoAurelio Well, "expected" to an extent, as I couldn't find any information about it before the actual shutdown. And BTW, is there a related task/discussion/you-name-it?

As for the priority, if the maintenance was planned, ideally this task should be just closed. However, other people may be confused after seeing that gerrit is down and this task will have answers for them. Plus this is really something that is to be addressed with the highest priority possible, so IMHO it should stay UBN - not that it makes much difference.

SRE (Operations) know about the problem and are working on it right now, a few folks commented about it on IRC. It's unknown yet when it will be back up.

(The error appears to have changed from “connection refused” to “connection timed out” now, though that’s probably not very significant.)

greg claimed this task.
greg subscribed.

Gerrit is back, sorry for the interruption.

greg removed greg as the assignee of this task.Mar 16 2019, 7:18 PM
thiemowmde subscribed.

I can't login to Gerrit any more. I created T218507, not knowing what was going on. Given this ticket doesn't explain much, I still don't know.

I don't think removing these comments is a good idea, as people subscribed to this task recieve the respective email notifications anyway and are left with even more questions than before. If the intention is to deny the claims of @JruwJN, I would consider it the better approach to just say so, if the intention is to keep this hidden for now, it simply is not working. So I don't think these deletions do any good.

I don't think removing these comments is a good idea, as people subscribed to this task recieve the respective email notifications anyway and are left with even more questions than before. If the intention is to deny the claims of @JruwJN, I would consider it the better approach to just say so, if the intention is to keep this hidden for now, it simply is not working. So I don't think these deletions do any good.

Ditto.

I think the intention is to (somewhat ?) limit the impact the vandal is trying to achieve (at least by removing the capability to link to those comments). While it's, as you point out due to the email notifications, impossible to fully mitigate that, and potentially causing other issues, I still consider it a prudent course of action. Other than that, please wait for a formal announcement (one that a link to will be posted to this task as well).

I do not really get how this is related to the impact of the vandal. as he basically states a single claim that would have to be denied or confirmed anyway at some point. I do not think we can link information about the incident and impact of the vandal here.

No need to keep random unconfirmed claims posted by an account with previously zero activity to spread FUD, jump guns, create confusion.
Please refer to https://lists.wikimedia.org/pipermail/wikitech-l/2019-March/091747.html instead. Thanks.

IMHO I don't think spreading FUD helps anyone. Wikimedia will let us all know in due time when they really know what happened.

Gerrit is not "down" any more, reopening this was a mistake. Sorry.