Page MenuHomePhabricator

ulsfo planned maintenance on 2016-05-11
Closed, ResolvedPublic


On Wednesday, 2016-05-11, we'll be depooling ulsfo from serving traffic. This is to ensure minimal (to no) user impact as I'll be on-site doing server maintenance. Once on-site work is complete, ulsfo will be placed back into service. Many of our systems were reporting high temperature readings for the CPUs, when the environmental temperature was acceptable. Investigation found the thermal paste on these systems had dried away to uselessness. As preventative measures, nearly half of the systems have had the thermal paste re-applied. The remainder will have this done on Wednesday.

This is related to T125205 & T119631.

Event Timeline

Mentioned in SAL [2016-05-10T23:21:12Z] <robh> disabling ulsfo via dns for onsite work tomorrow per T134831

Mentioned in SAL [2016-05-11T18:58:21Z] <robh> disregard ulsfo cp system icinga spam, onsite work for thermal paste per T134831

RobH added a subscriber: Joe.

This work was completed yesterday, and I returned ULSFO to service with

Had to learn conftool and how to repool varnish systems, which was overdue for me to learn anyhow. Thanks to @Joe for the ad hoc boot camp =]