Page MenuHomePhabricator

ulsfo pdu 1.22 replacement
Closed, ResolvedPublic

Description

Carlos with UL support called me, additionally we were emailed by their nagios system.

pdub-122.sf8 has issues on one of the two sides. We've depooled ulsfo as of 2016-11-21 @ 22:30 GMT. I advised UL support via email to begin their work an hour later than that.

The patchset to depool ulsfo is https://gerrit.wikimedia.org/r/#/c/322778/ it can be reverted at the end of the PDU replacement work.

Related Objects

StatusSubtypeAssignedTask
ResolvedRobH
DeclinedNone
ResolvedBBlack

Event Timeline

Since the PDU replacement, it turns out that cp4008 is the likely cause.

We replaced the PDU in your cab but it seems the culprit was either one of two power supplies that are out on your units----CP4008 & LVS4002.
These units are running on one side and need replacing.

Jon

When chatting with them shortly after the email (they called for followup), they advised that when the cp4008 was plugged back in, it didn't show any reaction to power being applied. This sounds like its PSU was already dead (perhaps causing the power distribution unit in the rack to fail). When they plugged back in lvs4002, it popped and they advised they could smell the burned power supply unit.

Both units will need replacing.

lvs4002 is an active lvs server, I'm not 100% confident pushing service back to ulsfo when the upload/maps/misc lvs system could go offline that easily (with minimal power supplies in operation.) We should likely fail over to lvs4004, but I'm not confident doing that solo.

I've dropped a text to @BBlack requesting advise. Since ulsfo is currently depooled in DNS, this is a high priority, but not unbreak now, type of situation.

@BBlack went ahead and depooled cp4008 from service and set lvs4002 as non-preferred over lvs4004 (so unless 4004 fails, it'll be primary and leave 2002 as secondary.)

the lvs preference had to be set at the router level.

dns has been set back to pool ulsfo

all fallout of this has been documented or resolved.