Page MenuHomePhabricator

ulsfo planned maintenance on 2016-05-11
Closed, ResolvedPublic

Description

On Wednesday, 2016-05-11, we'll be depooling ulsfo from serving traffic. This is to ensure minimal (to no) user impact as I'll be on-site doing server maintenance. Once on-site work is complete, ulsfo will be placed back into service. Many of our systems were reporting high temperature readings for the CPUs, when the environmental temperature was acceptable. Investigation found the thermal paste on these systems had dried away to uselessness. As preventative measures, nearly half of the systems have had the thermal paste re-applied. The remainder will have this done on Wednesday.

This is related to T125205 & T119631.

Event Timeline

RobH created this task.May 9 2016, 11:32 PM
Restricted Application added subscribers: Zppix, Southparkfan, Aklapper. · View Herald TranscriptMay 9 2016, 11:32 PM

Mentioned in SAL [2016-05-10T23:21:12Z] <robh> disabling ulsfo via dns for onsite work tomorrow per T134831

Mentioned in SAL [2016-05-11T18:58:21Z] <robh> disregard ulsfo cp system icinga spam, onsite work for thermal paste per T134831

RobH closed this task as Resolved.May 12 2016, 5:06 PM
RobH added a subscriber: Joe.

This work was completed yesterday, and I returned ULSFO to service with https://gerrit.wikimedia.org/r/#/c/288325/

Had to learn conftool and how to repool varnish systems, which was overdue for me to learn anyhow. Thanks to @Joe for the ad hoc boot camp =]