Comm Error: Backplane 0 on cirrussearch2091 (Row/Rack A7)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bking
	Apr 10 2025, 9:29 PM

Project Tags

Referenced Files

None

Subscribers

Description

Hello DC Ops,

I tried reimaging cirrussearch2091 and I got the following error:

Enumerating Boot options...
Enumerating Boot options... Done

iDRAC Settings:
HWC8010: The System Configuration Check operation resulted in the following
issue: Comm Error: Backplane 0.

UEFI0116: One or more boot drivers have reported issue(s).
Check the Driver Health Menu in Boot Manager for details.


Available Actions:
F1 to Continue and Retry Boot Order
F2 for System Setup (BIOS)
F10 for Lifecycle Controller
- Enable/Configure iDRAC
- Update Server Firmware
- Help Install an Operating System
F11 for Boot Manager

Are you able to take a look? There is some urgency, as we are in the middle of a migration (see parent ticket) and having more capacity would be helpful.

Thanks for your time! Ping me in IRC (inflatador) or Ryan Kemper (rkemper) if you have questions.

Related Objects
Search...

Status	Assigned	Task
Open	None	T370147 [Epic] Migration of the Elasticsearch 7.10 search cluster to OpenSearch
Resolved	bking	T388610 Migrate production Elastic clusters to Opensearch
Resolved	Jhancock.wm	T391639 Comm Error: Backplane 0 on cirrussearch2091 (Row/Rack A7)

Event Timeline

bking created this task.Apr 10 2025, 9:29 PM

Restricted Application edited projects, added: DC-Ops; removed: Patch-For-Review. · View Herald TranscriptApr 10 2025, 9:29 PM

Maintenance_bot added a project: SRE.Apr 10 2025, 10:29 PM

Gehel edited projects, added: Data-Platform-SRE (2025.04.12 - 2025.05.02); removed: Data-Platform-SRE (2025.03.22 - 2025.04.11).Apr 11 2025, 1:14 PM

Gehel edited projects, added: Discovery-Search (2025.04.11 - 2025.05.02); removed: Discovery-Search (2025.03.22 - 2025.04.11).Apr 11 2025, 1:26 PM

Jhancock.wm moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Apr 11 2025, 1:44 PM

reseated the cables to the backplane. no errors as of now. no os and didn't pxe. It should be fixed but please reopen and @ me if it happens again.

@bking

Gehel removed a project: Discovery-Search (2025.04.11 - 2025.05.02).Apr 15 2025, 8:36 AM

Gehel moved this task from Backlog - project to Done on the Data-Platform-SRE (2025.04.12 - 2025.05.02) board.Apr 16 2025, 2:52 PM

Thanks @Jhancock.wm ! Will try and reimage now.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.04.12 - 2025.05.02) board.Apr 17 2025, 3:52 PM

bking mentioned this in T388610: Migrate production Elastic clusters to Opensearch.Apr 21 2025, 10:49 PM

Jclark-ctr mentioned this in T398225: sessionstore1005 doesn't boot.Jul 2 2025, 2:05 AM

Jclark-ctr mentioned this in T398794: db1237 is not booting up.Jul 7 2025, 10:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1002 for host cirrussearch2091.codfw.wmnet with OS bullseye

bking mentioned this in T400099: cirrussearch2089 (A4) and cirrussearch2091 (A7) possible hardware issues.Jul 21 2025, 7:19 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1002 for host cirrussearch2091.codfw.wmnet with OS bullseye executed with errors:

cirrussearch2091 (FAIL)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced UEFI HTTP Boot for next reboot
- Host rebooted via Redfish
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cirrussearch2091.codfw.wmnet" to get a root shell, but depending on the failure this may not work.