Labservices1001 has died twice now, both times right after a flood of warnings about overheating.
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | fgiunchedi | T152340 labservices1001 down, suspected overheating | |||
| Resolved | • Cmjohnson | T154391 Replace fans (or paste) on labservices1001 |
Event Timeline
Comment Actions
Chris -- I'm not sure what the procedure is here. If you need to power down the machine for this we'll need to arrange an outage window -- otherwise, you can do this whenever :)
Comment Actions
Note to self: Before outage, silence icinga for labservices1001 and checker.tools.wmflabs.org
Comment Actions
I purchased some thermal paste today and the entire process should not take longer than 10mins.
Comment Actions
@Andrew Replaced the thermal paste on labservices1001....it didn't look dry and crusty so not 100% it will fix the issue. Leave the task open to monitor for a couple of weeks.
Comment Actions
The box has been solid since you worked on it. Unfortunately, the issue we seek to fix is VERY rare, so there won't really be any point where we declare it fixed for real. Might as well close the bug for now.
More good news, we're going to replace that hardware entirely in the next few months.