Page MenuHomePhabricator

No file system on toollabs, unable to login, web service broken
Closed, ResolvedPublic

Subscribers
Tokens
"The World Burns" token, awarded by Liuxinyu970226."The World Burns" token, awarded by Legoktm."The World Burns" token, awarded by MGChecker."The World Burns" token, awarded by Florian."The World Burns" token, awarded by Thibaut120094."The World Burns" token, awarded by Sjoerddebruin."The World Burns" token, awarded by APerson."The World Burns" token, awarded by revi."The World Burns" token, awarded by doctaxon."The World Burns" token, awarded by Romaine."The World Burns" token, awarded by DerHexer."The World Burns" token, awarded by Luke081515."The World Burns" token, awarded by Steinsplitter."The World Burns" token, awarded by Addshore.
Assigned To
Authored By
Multichill, Aug 30 2015

Description

multichill@tools-bastion-01:~/queries/wikidata$ ls
(nothing happens)

ssh tools-login.wmflabs.org

(just times out)

Web service on http://tools.wmflabs.org/ also broken. It gives 500 Internal Server Error (currently replaced by a "Our servers are currently experiencing a technical problem. " placeholder)

Event Timeline

Multichill raised the priority of this task from to Unbreak Now!.
Multichill updated the task description. (Show Details)
Multichill added a project: Toolforge.
Multichill added a subscriber: Multichill.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptAug 30 2015, 9:31 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Yuvi is working on this

The NFS server is down due to some kernel issues, we're working on it.

The last successful backup of tools is from 2015-08-30T01:59:35.787Z so at least we have a very recent backup

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptAug 30 2015, 10:53 AM
doctaxon set Security to None.Aug 30 2015, 10:55 AM
doctaxon added subscribers: coren, Andrew.
Multichill renamed this task from No file system on toollabs, unable to login to No file system on toollabs, unable to login, web service broken.Aug 30 2015, 11:00 AM
Multichill updated the task description. (Show Details)

I wonder why you haven't a redundant NFS server system not yet.

JanWMF added a subscriber: JanWMF.Aug 30 2015, 11:01 AM
Josve05a added a subscriber: Josve05a.
Addshore added a subscriber: Addshore.
Romaine rescinded a token.
Romaine awarded a token.
Emijrp added a subscriber: Emijrp.Aug 30 2015, 11:49 AM
revi awarded a token.Aug 30 2015, 12:00 PM
revi added a subscriber: revi.
NicoV added a subscriber: NicoV.Aug 30 2015, 12:20 PM
APerson added a subscriber: APerson.
jayvdb added a subscriber: jayvdb.Aug 30 2015, 1:10 PM
Florian added a subscriber: Florian.
Yogu added a subscriber: Yogu.Aug 30 2015, 1:16 PM

The NFS server and tool labs are back online.

Tool operators are getting a lot of failed jobs emails. All because of the outage. Emails should all have a timestamp that falls in the outage period.

MGChecker added a subscriber: MGChecker.
MGChecker awarded a token.
Emijrp removed a subscriber: Emijrp.Aug 30 2015, 4:33 PM
RP88 added a subscriber: RP88.Aug 31 2015, 11:50 PM
valhallasw closed this task as Resolved.Sep 2 2015, 9:54 AM
valhallasw claimed this task.

The initial issue was resolved sunday afternoon (CEST), but I forgot to close the task at that point.

Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptJun 7 2017, 6:47 PM