No file system on toollabs, unable to login, web service broken
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Multichill
	Aug 30 2015, 9:31 AM

Description

multichill@tools-bastion-01:~/queries/wikidata$ ls
(nothing happens)

ssh tools-login.wmflabs.org

(just times out)

Web service on http://tools.wmflabs.org/ also broken. It gives 500 Internal Server Error (currently replaced by a "Our servers are currently experiencing a technical problem. " placeholder)

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		valhallasw	T110827 No file system on toollabs, unable to login, web service broken
		Resolved		yuvipanda	T110832 labstore1002 not mounting all LVs after reboot

Event Timeline

Multichill created this task.Aug 30 2015, 9:31 AM

Multichill raised the priority of this task from to Unbreak Now!.

Multichill updated the task description. (Show Details)

Multichill added a project: Toolforge.

Multichill subscribed.

Restricted Application added a project: Cloud-Services. · View Herald TranscriptAug 30 2015, 9:31 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Yuvi is working on this

The NFS server is down due to some kernel issues, we're working on it.

https://phabricator.wikimedia.org/T110832 for NFS issues.

yuvipanda added a subtask: T110832: labstore1002 not mounting all LVs after reboot.Aug 30 2015, 10:23 AM

The last successful backup of tools is from 2015-08-30T01:59:35.787Z so at least we have a very recent backup

doctaxon subscribed.Aug 30 2015, 10:53 AM

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptAug 30 2015, 10:53 AM

doctaxon set Security to None.Aug 30 2015, 10:55 AM

doctaxon added subscribers: coren, Andrew.

Multichill renamed this task from No file system on toollabs, unable to login to No file system on toollabs, unable to login, web service broken.Aug 30 2015, 11:00 AM

Multichill updated the task description. (Show Details)

I wonder why you haven't a redundant NFS server system not yet.

JanWMF subscribed.Aug 30 2015, 11:01 AM

Steinsplitter subscribed.Aug 30 2015, 11:19 AM

Ricordisamoa subscribed.Aug 30 2015, 11:19 AM

Josve05a added a project: User-notice.Aug 30 2015, 11:19 AM

Josve05a subscribed.

Addshore awarded a token.Aug 30 2015, 11:20 AM

Addshore subscribed.

Steinsplitter awarded a token.Aug 30 2015, 11:20 AM

Luke081515 awarded a token.Aug 30 2015, 11:30 AM

DerHexer awarded a token.Aug 30 2015, 11:31 AM

Romaine awarded a token.Aug 30 2015, 11:34 AM

Romaine rescinded a token.

Romaine awarded a token.

doctaxon awarded a token.Aug 30 2015, 11:46 AM

Christian75 subscribed.Aug 30 2015, 11:47 AM

Emijrp subscribed.Aug 30 2015, 11:49 AM

revi awarded a token.Aug 30 2015, 12:00 PM

revi subscribed.

NicoV subscribed.Aug 30 2015, 12:20 PM

Enterprisey awarded a token.Aug 30 2015, 12:21 PM

Enterprisey subscribed.

Giftpflanze subscribed.Aug 30 2015, 12:38 PM

Pasleim subscribed.Aug 30 2015, 12:51 PM

Sjoerddebruin awarded a token.Aug 30 2015, 12:55 PM

Thibaut120094 awarded a token.Aug 30 2015, 1:09 PM

jayvdb subscribed.Aug 30 2015, 1:10 PM

Avicennasis subscribed.Aug 30 2015, 1:11 PM

Florian awarded a token.Aug 30 2015, 1:12 PM

Florian subscribed.

Yogu subscribed.Aug 30 2015, 1:16 PM

The NFS server and tool labs are back online.

Tool operators are getting a lot of failed jobs emails. All because of the outage. Emails should all have a timestamp that falls in the outage period.

MGChecker awarded a token.Aug 30 2015, 3:58 PM

MGChecker subscribed.

MGChecker rescinded a token.Aug 30 2015, 4:01 PM

MGChecker awarded a token.

jeremyb-phone added a subscriber: jeremyb.Aug 30 2015, 4:32 PM

Emijrp unsubscribed.Aug 30 2015, 4:33 PM

Krenair subscribed.Aug 31 2015, 2:07 AM

Legoktm awarded a token.Aug 31 2015, 7:41 AM

RP88 subscribed.Aug 31 2015, 11:50 PM

Johan moved this task from To Triage to Announce in next Tech/News on the User-notice board.Sep 2 2015, 9:46 AM