Page MenuHomePhabricator

gerrit.wikimedia.org is down
Closed, ResolvedPublic

Tokens
"Cup of Joe" token, awarded by Mvolz."The World Burns" token, awarded by MGChecker."The World Burns" token, awarded by Base."Like" token, awarded by Ammarpad."The World Burns" token, awarded by matej_suchanek.
Assigned To
None
Authored By
Daimona, Mar 16 2019

Description

What it says on the tin. Trying to open any page results in a connection refused error. Already checked that it's not only me, and icinga has a few errors about that from 10 minutes ago.

Event Timeline

Daimona created this task.Mar 16 2019, 10:13 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 16 2019, 10:13 AM
Daimona triaged this task as Unbreak Now! priority.Mar 16 2019, 10:14 AM
Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptMar 16 2019, 10:14 AM
<chasemp> !log stop apache on cobalt for maintenance
Mmarx added a subscriber: Mmarx.Mar 16 2019, 11:09 AM
Base awarded a token.Mar 16 2019, 11:37 AM
Base added a subscriber: Base.

Well if gerrit is on maintenance and apache has been disconnected then this downtime is to be expected :-)

Given that this is not an involuntary outage, shall we degrade from UBN?

@MarcoAurelio Well, "expected" to an extent, as I couldn't find any information about it before the actual shutdown. And BTW, is there a related task/discussion/you-name-it?

As for the priority, if the maintenance was planned, ideally this task should be just closed. However, other people may be confused after seeing that gerrit is down and this task will have answers for them. Plus this is really something that is to be addressed with the highest priority possible, so IMHO it should stay UBN - not that it makes much difference.

MGChecker added a subscriber: MGChecker.

SRE (Operations) know about the problem and are working on it right now, a few folks commented about it on IRC. It's unknown yet when it will be back up.

Mvolz awarded a token.Mar 16 2019, 3:07 PM
Mvolz added a subscriber: Mvolz.
Dalba added a subscriber: Dalba.Mar 16 2019, 3:30 PM
Tassu added a subscriber: Tassu.Mar 16 2019, 3:59 PM
abian added a subscriber: abian.Mar 16 2019, 4:30 PM
Zppix added a subscriber: Zppix.Mar 16 2019, 5:47 PM

(The error appears to have changed from “connection refused” to “connection timed out” now, though that’s probably not very significant.)

greg closed this task as Resolved.Mar 16 2019, 7:18 PM
greg claimed this task.
greg added a subscriber: greg.

Gerrit is back, sorry for the interruption.

Restricted Application added a project: User-greg. · View Herald TranscriptMar 16 2019, 7:18 PM
greg removed greg as the assignee of this task.Mar 16 2019, 7:18 PM
MaxSem added a subscriber: MaxSem.Mar 16 2019, 9:58 PM
This comment was removed by Ladsgroup.
thiemowmde reopened this task as Open.Mar 17 2019, 11:25 AM
thiemowmde added a subscriber: thiemowmde.

I can't login to Gerrit any more. I created T218507, not knowing what was going on. Given this ticket doesn't explain much, I still don't know.

This comment was removed by Ladsgroup.

I don't think removing these comments is a good idea, as people subscribed to this task recieve the respective email notifications anyway and are left with even more questions than before. If the intention is to deny the claims of @JruwJN, I would consider it the better approach to just say so, if the intention is to keep this hidden for now, it simply is not working. So I don't think these deletions do any good.

I don't think removing these comments is a good idea, as people subscribed to this task recieve the respective email notifications anyway and are left with even more questions than before. If the intention is to deny the claims of @JruwJN, I would consider it the better approach to just say so, if the intention is to keep this hidden for now, it simply is not working. So I don't think these deletions do any good.

Ditto.

akosiaris added a comment.EditedMar 17 2019, 12:44 PM

I think the intention is to (somewhat ?) limit the impact the vandal is trying to achieve (at least by removing the capability to link to those comments). While it's, as you point out due to the email notifications, impossible to fully mitigate that, and potentially causing other issues, I still consider it a prudent course of action. Other than that, please wait for a formal announcement (one that a link to will be posted to this task as well).

I do not really get how this is related to the impact of the vandal. as he basically states a single claim that would have to be denied or confirmed anyway at some point. I do not think we can link information about the incident and impact of the vandal here.

No need to keep random unconfirmed claims posted by an account with previously zero activity to spread FUD, jump guns, create confusion.
Please refer to https://lists.wikimedia.org/pipermail/wikitech-l/2019-March/091747.html instead. Thanks.

IMHO I don't think spreading FUD helps anyone. Wikimedia will let us all know in due time when they really know what happened.

Tassu removed a subscriber: Tassu.Mar 17 2019, 2:02 PM
JJMC89 added a subscriber: JJMC89.Mar 17 2019, 5:19 PM
thiemowmde closed this task as Resolved.Mar 17 2019, 7:36 PM

Gerrit is not "down" any more, reopening this was a mistake. Sorry.

Risker added a subscriber: Risker.Mar 17 2019, 8:07 PM
Mvolz rescinded a token.Mar 18 2019, 10:04 AM
Mvolz awarded a token.
debt added a subscriber: debt.Mar 18 2019, 4:08 PM