Maniphest T152368

labservices1001 crashed and sent no pages
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Andrew
	Dec 5 2016, 5:12 AM

Project Tags

Referenced Files

None

Subscribers

Description

Labservices1001 died (T152340) and I found out about it from the toolschecker. If the toolschecker had failed over properly between DNS hosts, we would not have know about this outage AT ALL until new VM creation started failing.

That's crazy. Why didn't a down host cause icinga to light up like a Christmas tree? Or, did it and I'm just not subscribed to those pages somehow?

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Designate: page if services go down.	Andrew Bogott	operations/puppet	production	+7 -0
	Page if a labs dns server stops responding	Andrew Bogott	operations/puppet	production	+1 -0

Customize query in gerrit

Related Objects

Mentioned Here: T152340: labservices1001 down, suspected overheating

Event Timeline

Andrew created this task.Dec 5 2016, 5:12 AM

Restricted Application added a project: Cloud-Services. · View Herald TranscriptDec 5 2016, 5:12 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Paladox subscribed.Dec 5 2016, 7:34 AM

Peachey88 added a project: observability.Dec 5 2016, 7:39 AM

fgiunchedi added a project: Wikimedia-Incident.Dec 5 2016, 7:49 PM

Krenair subscribed.Dec 5 2016, 7:51 PM

Change 325358 had a related patch set uploaded (by Andrew Bogott):
Page if a labs dns server stops responding

https://gerrit.wikimedia.org/r/325358

Change 325359 had a related patch set uploaded (by Andrew Bogott):
Designate: page if services go down.

https://gerrit.wikimedia.org/r/325359

greg moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.Dec 5 2016, 10:11 PM

Change 325358 abandoned by Andrew Bogott:
Page if a labs dns server stops responding

https://gerrit.wikimedia.org/r/325358

Change 325359 merged by Andrew Bogott:
Designate: page if services go down.

https://gerrit.wikimedia.org/r/325359

Andrew closed this task as Resolved.Dec 8 2016, 8:33 PM

Krinkle edited projects, added: Sustainability (Incident Followup); removed: Wikimedia-Incident, Cloud-VPS.Apr 28 2020, 9:50 PM