db2135 (C6) lost power supply redundancy
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	jcrespo
	Aug 5 2022, 6:56 AM

Description

@Papaul can you check the status of db2135 power supplies. It lives in C6 and we just got this alert:

<+icinga-wm> PROBLEM - IPMI Sensor Status on db2135 is CRITICAL: Sensor Type(s) Temperature, Power_Supply Status: Critical [PS Redundancy = Critical, Status = Critical] https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Power_Supply_Failures

Thanks!

Details

	Subject	Repo	Branch	Lines +/-
	mariadb: Reenable notifications for db2135	operations/puppet	production	+0 -1

Customize query in gerrit

Related Objects

Mentioned In: T314559: es2021 (B3) lost power supply redundancy
Mentioned Here: P32292 dbctl commit (dc=all): 'Depool hosts with fragile power supply (T314559 T314628)'
T314559: es2021 (B3) lost power supply redundancy

Event Timeline

jcrespo created this task.Aug 5 2022, 6:56 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 5 2022, 6:56 AM

Task similar to T314559, I wonder if just a loose cable or the power supplies died.

Maintenance_bot added a project: SRE.Aug 5 2022, 7:29 AM

It's depooled, I will disable notifications for this and T314559: es2021 (B3) lost power supply redundancy

Mentioned in SAL (#wikimedia-operations) [2022-08-05T13:27:09Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depool hosts with fragile power supply (T314559 T314628)', diff saved to https://phabricator.wikimedia.org/P32292 and previous config saved to /var/cache/conftool/dbconfig/20220805-132709-ladsgroup.json