Page MenuHomePhabricator

kafka-main2008 CPU 2 MEMABCD VPP PG voltage is outside of range.
Closed, ResolvedPublic

Description

kafka-main2008 was shut down for reimage and does not come back up:

racadm>> getsel
-------------------------------------------------------------------------------
Record:      9
Date/Time:   03/26/2025 14:34:34
Source:      system
Severity:    Ok
Description: Power supply 2 is operating normally.
-------------------------------------------------------------------------------
Record:      10
Date/Time:   06/09/2026 17:06:22
Source:      system
Severity:    Critical
Description: CPU 2 MEMABCD VPP PG voltage is outside of range.
-------------------------------------------------------------------------------
Record:      11
Date/Time:   06/09/2026 17:06:28
Source:      system
Severity:    Critical
Description: The system board Pfault fail-safe voltage is outside of range.
-------------------------------------------------------------------------------

Tried to powerup/powercycle via racadm again did not make it power up:

racadm>>serveraction powercycle
Server power operation initiated successfully
racadm>>serveraction powerstatus
Server power status: OFF
racadm>>

Failure Notes

https://netbox.wikimedia.org/dcim/devices/5238/
In warranty until April 30, 2027
CPU 2 MEMABCD VPP PG voltage is outside of range.
The system board Pfault fail-safe voltage is outside of range.

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2026-06-09T17:43:38Z] <jayme> kafka-main2008 is down due to hardware failure T428654

RobH triaged this task as High priority.Jun 9 2026, 5:46 PM
RobH updated the task description. (Show Details)
RobH updated the task description. (Show Details)
RobH moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.

unplugged the PSUs and drained the flea power. had it PXE boot at Jayme's request. Reboot fixed it, but if it happens again, please let us know and we'll work with Dell to find the cause and replace it.

JMeybohm claimed this task.

unplugged the PSUs and drained the flea power. had it PXE boot at Jayme's request. Reboot fixed it, but if it happens again, please let us know and we'll work with Dell to find the cause and replace it.

The reimage we where trying to do did complete successfully and the system booted normally after the initial puppet run. Thanks again for your help and quick response!