$ sudo grep mcelog syslog
Dec  4 01:56:22 labservices1001 mcelog: mcelog read: Function not implemented
Dec  4 01:56:22 labservices1001 mcelog: Processor 1 heated above trip temperature. Throttling enabled.
Dec  4 01:56:22 labservices1001 mcelog: Please check your system cooling. Performance will be impacted
Dec  4 01:56:22 labservices1001 mcelog: Processor 1 below trip temperature. Throttling disabled
Dec  4 01:58:52 labservices1001 mcelog: Processor 3 heated above trip temperature. Throttling enabled.
Dec  4 01:58:52 labservices1001 mcelog: Please check your system cooling. Performance will be impacted
Dec  4 01:58:52 labservices1001 mcelog: Processor 3 below trip temperature. Throttling disabled
Dec  4 02:06:22 labservices1001 mcelog: Processor 3 heated above trip temperature. Throttling enabled.
Dec  4 02:06:22 labservices1001 mcelog: Please check your system cooling. Performance will be impacted
Dec  4 02:06:22 labservices1001 mcelog: Processor 3 below trip temperature. Throttling disabled
Dec  4 02:08:52 labservices1001 mcelog: Processor 3 heated above trip temperature. Throttling enabled.
Dec  4 02:08:52 labservices1001 mcelog: Please check your system cooling. Performance will be impacted
Dec  4 02:08:52 labservices1001 mcelog: Processor 3 below trip temperature. Throttling disabled
Dec  4 02:12:37 labservices1001 mcelog: Processor 1 heated above trip temperature. Throttling enabled.
Dec  4 02:12:37 labservices1001 mcelog: Please check your system cooling. Performance will be impacted
Dec  4 02:12:37 labservices1001 mcelog: Processor 2 heated above trip temperature. Throttling enabled.
Dec  4 02:12:37 labservices1001 mcelog: Please check your system cooling. Performance will be impacted
Dec  4 02:42:00 labservices1001 mcelog: failed to prefill DIMM database from DMI data

And before that labservices1001 saw high disk I/O starting at around 1.00 UTC which might have been a trigger

https://grafana.wikimedia.org/dashboard/db/prometheus-machine-stats?var-server=labservices1001%3A9100&var-datasource=eqiad%20prometheus%2Fops&from=1480812690403&to=1480818733445

Note that during this time dns from labs instances seemed fine, why toolschecker failed needs investigation.

This task is to track the suspected overheating / hw problem

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		fgiunchedi	T152340 labservices1001 down, suspected overheating
		Resolved		• Cmjohnson	T154391 Replace fans (or paste) on labservices1001

Event Timeline

fgiunchedi created this task.Dec 4 2016, 2:57 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 4 2016, 2:57 AM

fgiunchedi assigned this task to Andrew.Dec 4 2016, 2:57 AM

fgiunchedi added projects: SRE, Cloud-Services.

Krenair added a project: Cloud-VPS.Dec 4 2016, 2:57 AM

Krenair subscribed.Dec 4 2016, 3:01 AM

Note that during this time dns from labs instances seemed fine, why toolschecker failed needs investigation

Well, it wasn't all fine. CI tests were failing with stuff like:

02:21:30 git.exc.GitCommandError: 'git remote prune --dry-run origin' returned with exit code 128
02:21:30 stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/p/mediawiki/core/': Could not resolve host: gerrit.wikimedia.org'

Paladox subscribed.Dec 4 2016, 11:23 AM

Andrew mentioned this in T152368: labservices1001 crashed and sent no pages.Dec 5 2016, 5:12 AM

Andrew mentioned this in T152370: Do contintcloud and other CI boxes know about labs-ns1?.Dec 5 2016, 5:18 AM

fgiunchedi added a project: Wikimedia-Incident.Dec 5 2016, 7:49 PM

This is now back up, yes? :) Incident report filed at https://wikitech.wikimedia.org/wiki/Incident_documentation/20161204-labservices1001 Closable? Follow-ups all filed and such?

@greg yeah now back up! this task is one of the followups though, I'll clarify it a bit

fgiunchedi renamed this task from labservices1001 down to labservices1001 down, suspected overheating.Dec 6 2016, 6:57 PM

fgiunchedi updated the task description. (Show Details)

Andrew removed Andrew as the assignee of this task.Dec 22 2016, 12:11 AM

Andrew subscribed.

Andrew created subtask T154391: Replace fans (or paste) on labservices1001.Jan 1 2017, 2:41 AM

A few things that happen that should not when labservices1001 dies (we do not see these same failures when labservices1002 is down):

Puppet runs fail (in tools at least) attempting to resolve tools-redis-1001.tools.eqiad.wmflabs
Tools checker checks /all/ go offline it seems
CI tests (or some significant subset or them) fail to run adequately (see https://phabricator.wikimedia.org/T152340#2845120)

It's probable this is all due to hardcoded DNS resolution or some failure in this category.

• chasemp triaged this task as High priority.Jan 3 2017, 2:21 PM

tom29739 subscribed.Jan 3 2017, 3:36 PM

greg moved this task from Active investigation to Active Situation on the Wikimedia-Incident board.Jan 5 2017, 5:12 PM

greg moved this task from Active Situation to Follow-up prevention on the Wikimedia-Incident board.

Andrew closed subtask T154391: Replace fans (or paste) on labservices1001 as Resolved.Jan 23 2017, 3:59 PM