Change Details

Today labservices1001 went down and toolschecker complained about all tests failing. The failure itself seems hardware related, host down was at 2:17 ``` $ sudo grep mcelog syslog Dec 4 01:56:22 labservices1001 mcelog: mcelog read: Function not implemented Dec 4 01:56:22 labservices1001 mcelog: Processor 1 heated above trip temperature. Throttling enabled. Dec 4 01:56:22 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 01:56:22 labservices1001 mcelog: Processor 1 below trip temperature. Throttling disabled Dec 4 01:58:52 labservices1001 mcelog: Processor 3 heated above trip temperature. Throttling enabled. Dec 4 01:58:52 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 01:58:52 labservices1001 mcelog: Processor 3 below trip temperature. Throttling disabled Dec 4 02:06:22 labservices1001 mcelog: Processor 3 heated above trip temperature. Throttling enabled. Dec 4 02:06:22 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 02:06:22 labservices1001 mcelog: Processor 3 below trip temperature. Throttling disabled Dec 4 02:08:52 labservices1001 mcelog: Processor 3 heated above trip temperature. Throttling enabled. Dec 4 02:08:52 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 02:08:52 labservices1001 mcelog: Processor 3 below trip temperature. Throttling disabled Dec 4 02:12:37 labservices1001 mcelog: Processor 1 heated above trip temperature. Throttling enabled. Dec 4 02:12:37 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 02:12:37 labservices1001 mcelog: Processor 2 heated above trip temperature. Throttling enabled. Dec 4 02:12:37 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 02:42:00 labservices1001 mcelog: failed to prefill DIMM database from DMI data ``` And before that labservices1001 saw high disk I/O starting at around 1.00 UTC which might have been a trigger https://grafana.wikimedia.org/dashboard/db/prometheus-machine-stats?var-server=labservices1001%3A9100&var-datasource=eqiad%20prometheus%2Fops&from=1480812690403&to=1480818733445 Note that during this time dns from labs instances seemed fine, why toolschecker failed needs investigation

Today labservices1001 went down and toolschecker complained about all tests failing. The failure itself seems hardware related, host down was at 2:17 ``` $ sudo grep mcelog syslog Dec 4 01:56:22 labservices1001 mcelog: mcelog read: Function not implemented Dec 4 01:56:22 labservices1001 mcelog: Processor 1 heated above trip temperature. Throttling enabled. Dec 4 01:56:22 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 01:56:22 labservices1001 mcelog: Processor 1 below trip temperature. Throttling disabled Dec 4 01:58:52 labservices1001 mcelog: Processor 3 heated above trip temperature. Throttling enabled. Dec 4 01:58:52 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 01:58:52 labservices1001 mcelog: Processor 3 below trip temperature. Throttling disabled Dec 4 02:06:22 labservices1001 mcelog: Processor 3 heated above trip temperature. Throttling enabled. Dec 4 02:06:22 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 02:06:22 labservices1001 mcelog: Processor 3 below trip temperature. Throttling disabled Dec 4 02:08:52 labservices1001 mcelog: Processor 3 heated above trip temperature. Throttling enabled. Dec 4 02:08:52 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 02:08:52 labservices1001 mcelog: Processor 3 below trip temperature. Throttling disabled Dec 4 02:12:37 labservices1001 mcelog: Processor 1 heated above trip temperature. Throttling enabled. Dec 4 02:12:37 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 02:12:37 labservices1001 mcelog: Processor 2 heated above trip temperature. Throttling enabled. Dec 4 02:12:37 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 02:42:00 labservices1001 mcelog: failed to prefill DIMM database from DMI data ``` And before that labservices1001 saw high disk I/O starting at around 1.00 UTC which might have been a trigger https://grafana.wikimedia.org/dashboard/db/prometheus-machine-stats?var-server=labservices1001%3A9100&var-datasource=eqiad%20prometheus%2Fops&from=1480812690403&to=1480818733445 Note that during this time dns from labs instances seemed fine, why toolschecker failed needs investigation. This task is to track the suspected overheating / hw problem

Today labservices1001 went down and toolschecker complained about all tests failing. The failure itself seems hardware related, host down was at 2:17 ``` $ sudo grep mcelog syslog Dec 4 01:56:22 labservices1001 mcelog: mcelog read: Function not implemented Dec 4 01:56:22 labservices1001 mcelog: Processor 1 heated above trip temperature. Throttling enabled. Dec 4 01:56:22 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 01:56:22 labservices1001 mcelog: Processor 1 below trip temperature. Throttling disabled Dec 4 01:58:52 labservices1001 mcelog: Processor 3 heated above trip temperature. Throttling enabled. Dec 4 01:58:52 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 01:58:52 labservices1001 mcelog: Processor 3 below trip temperature. Throttling disabled Dec 4 02:06:22 labservices1001 mcelog: Processor 3 heated above trip temperature. Throttling enabled. Dec 4 02:06:22 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 02:06:22 labservices1001 mcelog: Processor 3 below trip temperature. Throttling disabled Dec 4 02:08:52 labservices1001 mcelog: Processor 3 heated above trip temperature. Throttling enabled. Dec 4 02:08:52 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 02:08:52 labservices1001 mcelog: Processor 3 below trip temperature. Throttling disabled Dec 4 02:12:37 labservices1001 mcelog: Processor 1 heated above trip temperature. Throttling enabled. Dec 4 02:12:37 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 02:12:37 labservices1001 mcelog: Processor 2 heated above trip temperature. Throttling enabled. Dec 4 02:12:37 labservices1001 mcelog: Please check your system cooling. Performance will be impacted Dec 4 02:42:00 labservices1001 mcelog: failed to prefill DIMM database from DMI data ``` And before that labservices1001 saw high disk I/O starting at around 1.00 UTC which might have been a trigger https://grafana.wikimedia.org/dashboard/db/prometheus-machine-stats?var-server=labservices1001%3A9100&var-datasource=eqiad%20prometheus%2Fops&from=1480812690403&to=1480818733445 Note that during this time dns from labs instances seemed fine, why toolschecker failed needs investigation. This task is to track the suspected overheating / hw problem