@Cmjohnson ping :) Can we apply some thermal paste to one host as test?

In T132256 Chris applied the thermal paste to analytics1039 and the alerts stopped, so it seems the right way to go. Since the number of appservers is big we might need to schedule it properly to avoid 8 hours of thermal paste per day for Chris :)

mw1227 has been alerting over the weekend of high load, I depooled it and noticed it was on the list of machines with temperature overheating as well, so likely related.

Mentioned in SAL (#wikimedia-operations) [2018-02-19T08:11:16Z] <godog> repool mw1227 - T149287

In T149287#3963590, @fgiunchedi wrote:

mw1227 has been alerting over the weekend of high load, I depooled it and noticed it was on the list of machines with temperature overheating as well, so likely related.

Turns out that's not the case, likely hhvm was locking up instead

Most of the servers are decommissioned. Are you still have problems with

mw1221.eqiad.wmnet
mw1222.eqiad.wmnet
mw1225.eqiad.wmnet
mw1226.eqiad.wmnet
mw1227.eqiad.wmnet
mw1229.eqiad.wmnet
mw1230.eqiad.wmnet
mw1231.eqiad.wmnet
mw1232.eqiad.wmnet
mw1233.eqiad.wmnet
mw1234.eqiad.wmnet
mw1236.eqiad.wmnet
mw1237.eqiad.wmnet
mw1238.eqiad.wmnet
mw1240.eqiad.wmnet
mw1241.eqiad.wmnet
mw1242.eqiad.wmnet
mw1244.eqiad.wmnet
mw1246.eqiad.wmnet
mw1255.eqiad.wmnet

mw1221, mw1230 and mw1235 are fine, the others are still showing the mentioned symptoms.

Thanks Moritz. I have a procurement task for more thermal paste. Once it arrives, we can schedule a time to take care of these.
procurement task https://phabricator.wikimedia.org/T198326

@Cmjohnson Thanks, sounds good.

@Cmjohnson : Per the procurement task, thermal paste is now available?

We observed overheating symptoms on the following machines today:
mw[1221-1227,1229,1231-1235,1238,1240-1248,1250-1251,1253,1255].eqiad.wmnet

Many of these machines are always running hot -- ambient temps of 85C or more, even when only lightly loaded.

I whipped up a quick Grafana graph of some of the temperatures:
https://grafana.wikimedia.org/dashboard/db/xxxx-cdanis-test?panelId=1&fullscreen&orgId=1&from=now-7d&to=now

please add mw1252 to the list (T236190)

mw1267 was showing temperature issues today:

https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&from=now-2d&to=now&fullscreen&panelId=25&var-server=mw1267&var-datasource=eqiad%20prometheus%2Fops&var-cluster=appserver

We lowered its weight in conftool from 30 to 25 and restarted php-fpm. Load went down from about 55 to 15 shortly after.