Page MenuHomePhabricator
Create Task
Maniphest T420416

hw troubleshooting: Comm Error: Backplane 0 on an-worker1172.eqiad.wmnet
Closed, ResolvedPublic
Actions

Assigned To
Jclark-ctr
Authored By
RKemper
Mar 17 2026, 9:51 PM
Project Tags
Referenced Files
F73241550: image.png
Mar 20 2026, 9:34 AM
F73036246: Screenshot 2026-03-18 at 2.35.47 PM.png
Mar 18 2026, 6:36 PM
F72974315: image.png
Mar 17 2026, 11:50 PM
F72974098: image.png
Mar 17 2026, 11:50 PM
F72973897: image.png
Mar 17 2026, 11:50 PM
Subscribers
Aklapper
BTullis
Gehel
Jclark-ctr
RKemper

Description

FQDN: an-worker1172.eqiad.wmnet

Netbox: Set to Failed (https://netbox.wikimedia.org/dcim/devices/5037/)

Priority: Medium

Host can be worked on at will (it's already down)

Timeline
  • 2026-03-03 11:13 UTC — btullis ran sre.hosts.reboot-single (START)
  • 2026-03-03 12:08 UTC — cookbook returned END (PASS) after ~55 minutes, meaning the host came back up far enough to satisfy health checks
  • 2026-03-06 (or earlier) — host found SSH-unreachable; given the below backplane error, it prob very quickly fell back into failed state after the 2026-03-03 reboot
Record:      26
Date/Time:   03/03/2026 11:17:38
Source:      system
Severity:    Critical
Description: The System Configuration Check operation resulted in the following issue: Comm Error: Backplane 0.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Temporarily set an-worker1172 into insetup modeBtullisoperations/puppetproduction+5 -1
Update the partman recipe for an-worker1172Btullisoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects

Event Timeline

RKemper created this task.Mar 17 2026, 9:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 17 2026, 9:52 PM
RKemper renamed this task from an-worker1172.eqiad.wmnet unreachable since 2026-03-03 to hw troubleshooting: Comm Error: Backplane 0 on an-worker1172.eqiad.wmnet.Mar 17 2026, 10:02 PM
RKemper reassigned this task from RKemper to Jclark-ctr.
RKemper added projects: DC-Ops, ops-eqiad.
RKemper updated the task description. (Show Details)
RKemper moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.

Switched this to a HW failure ticket, given racadm getsel revealed a backplane issue

RKemper moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.Mar 17 2026, 10:03 PM
RKemper added a project: Essential-Work.
RKemper triaged this task as Medium priority.Mar 17 2026, 10:06 PM
RKemper updated the task description. (Show Details)
RKemper updated the task description. (Show Details)
Maintenance_bot added a project: SRE.Mar 17 2026, 10:29 PM
BTullis subscribed.Mar 17 2026, 11:50 PM

I confirmed that the backplane was giving this error.

image.png (2,388×1,245 px, 619 KB)

But then I tried cold resetting the BMC.

btullis@cumin1003:~$ sudo ipmitool -I lanplus -H "an-worker1172.mgmt.eqiad.wmnet" -U root -E shell
Unable to read password from environment
Password: 
ipmitool> bmc reset cold
Sent cold reset command to MC

Initially, the results looked good after reconnecting...

image.png (2,388×1,245 px, 455 KB)

But upon powering up, the error came back again.
image.png (2,388×1,245 px, 548 KB)

Jclark-ctr added a comment.Mar 18 2026, 6:36 PM

@BTullis
Performed firmware update on backplane seems to of cleared but is not booting might require reimage.

Screenshot 2026-03-18 at 2.35.47 PM.png (2,094×1,330 px, 243 KB)

Jclark-ctr closed this task as Resolved.Mar 19 2026, 6:52 PM
Gehel reopened this task as Open.Mar 20 2026, 9:13 AM
Gehel moved this task from Blocked/Waiting to In Progress on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.
Gehel subscribed.

Re-opening and moving to in progress to finalize reimage.

BTullis claimed this task.Mar 20 2026, 9:15 AM
BTullis added a subscriber: Jclark-ctr.
ops-monitoring-bot added a comment.Mar 20 2026, 9:17 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 20 2026, 9:26 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1172 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1172.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
gerritbot added a comment.Mar 20 2026, 9:31 AM

Change #1256270 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Temporarily set an-worker1172 into insetup mode

https://gerrit.wikimedia.org/r/1256270

gerritbot added a project: Patch-For-Review.Mar 20 2026, 9:31 AM
BTullis added a comment.Mar 20 2026, 9:34 AM

The first reimage failed because of a partman issue.

image.png (899×536 px, 76 KB)

I'll put the host into insetup mode to carry out the reimage.

gerritbot added a comment.Mar 20 2026, 10:00 AM

Change #1256270 merged by Btullis:

[operations/puppet@production] Temporarily set an-worker1172 into insetup mode

https://gerrit.wikimedia.org/r/1256270

Maintenance_bot removed a project: Patch-For-Review.Mar 20 2026, 10:31 AM
ops-monitoring-bot added a comment.Mar 20 2026, 11:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 20 2026, 11:42 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1172 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1172.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mar 20 2026, 11:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 20 2026, 4:09 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1172 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1172.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
gerritbot added a comment.Mar 20 2026, 4:14 PM

Change #1256426 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update the partman recipe for an-worker1172

https://gerrit.wikimedia.org/r/1256426

gerritbot added a project: Patch-For-Review.Mar 20 2026, 4:14 PM
gerritbot added a comment.Mar 20 2026, 4:18 PM

Change #1256426 merged by Btullis:

[operations/puppet@production] Update the partman recipe for an-worker1172

https://gerrit.wikimedia.org/r/1256426

Maintenance_bot removed a project: Patch-For-Review.Mar 20 2026, 4:30 PM
ops-monitoring-bot added a comment.Mar 20 2026, 4:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 20 2026, 4:54 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1172 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1172.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mar 20 2026, 4:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2026, 9:29 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1172 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1172.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mar 23 2026, 9:29 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2026, 9:39 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1172 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1172.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mar 23 2026, 9:40 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2026, 10:09 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1172 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console an-worker1172.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mar 23 2026, 1:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2026, 2:30 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host an-worker1172.eqiad.wmnet with OS bullseye completed:

  • an-worker1172 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202603231359_jclark_1496331_an-worker1172.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status failed -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr added a comment.Mar 23 2026, 2:32 PM

@BTullis , I was able to reimage it. The an-workers always seem to have issues when the HDD virtual drives are present during imaging. Would you be able to add them back in with a single command instead of me having to do each one manually?

BTullis closed this task as Resolved.Mar 23 2026, 4:54 PM
BTullis moved this task from In Progress to Done on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.

I belive that this is now fixed. Thanks @Jclark-ctr .

There is an outstanding puppet error, but I believe that this is unrelated to the hardware issue, so I'll close this ticket.

Jclark-ctr claimed this task.Mar 24 2026, 12:17 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.Mar 27 2026, 8:42 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits