Unresponsive management for an-worker1147.mgmt:22
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	phaultfinder
	Jul 28 2026, 8:38 AM

Project Tags

Referenced Files

None

Subscribers

Description

Common information

dashboard: TODO
description: The management interface at an-worker1147.mgmt:22 has been unresponsive for multiple hours.
runbook: https://wikitech.wikimedia.org/wiki/Management_Interfaces#Reset_the_management_card
summary: Unresponsive management for an-worker1147.mgmt:22

alertname: ManagementSSHDown
instance: an-worker1147.mgmt:22
job: probes/mgmt
module: ssh_banner
prometheus: ops
rack: E1
severity: task
site: eqiad
source: prometheus
team: dcops

Firing alerts

dashboard: TODO
description: The management interface at an-worker1147.mgmt:22 has been unresponsive for multiple hours.
runbook: https://wikitech.wikimedia.org/wiki/Management_Interfaces#Reset_the_management_card
summary: Unresponsive management for an-worker1147.mgmt:22
alertname: ManagementSSHDown
instance: an-worker1147.mgmt:22
job: probes/mgmt
module: ssh_banner
prometheus: ops
rack: E1
severity: task
site: eqiad
source: prometheus
team: dcops
Source

Related Objects
Search...

Status	Assigned	Task
Resolved	brouberol	T401692 EPIC: Migrate Data Platform SRE-owned hosts to Bookworm or later
Resolved	BTullis	T434494 Migrate production hadoop cluster to bookworm
Resolved	BTullis	T433348 Unresponsive management for an-worker1147.mgmt:22
Resolved	VRiley-WMF	T435390 Physical connection testing

Event Timeline

phaultfinder created this task.Jul 28 2026, 8:38 AM

Restricted Application added a project: DC-Ops. · View Herald TranscriptJul 28 2026, 8:38 AM

Maintenance_bot added a project: SRE.Jul 28 2026, 9:29 AM

VRiley-WMF moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Jul 29 2026, 3:00 PM

VRiley-WMF claimed this task.Aug 3 2026, 5:39 PM

@bking Hey, I wanted to reach out about this server. Is there a good time to reboot this? We're having some issues with the iDRAC and I'd like to try to power drain it. We can do it anytime today or tomorrow.

bking edited projects, added: Data-Platform-SRE (2026-07-03 - 2026-07-31); removed: SRE.Aug 4 2026, 3:57 PM

@VRiley-WMF I don't interact with the an-worker servers much, so I've asked my team in Slack. We'll get back to you when have more info.

Thank you! I'm ready whenever they are ready to take it down.

Gehel edited projects, added: Data-Platform-SRE (2026-08-07 - 2026-08-28); removed: Data-Platform-SRE (2026-07-03 - 2026-07-31).Tue, Aug 11, 6:52 AM

@bking I don't know if there has been an update on this? Or if there is someone else to reach out to, I'd be more than happy to. I did check slack and it didn't seem like anyone responded unless I missed it?

Let me know, thanks!

@VRiley-WMF We discussed this internally, you're free to proceed whenever. These hosts aren't lvs-backed so no depool concerns there, and the cluster should weather the single node loss without issue

Hi, sorry for the delay in getting back to you about this.

I can't get access to the management interface, either. I tried SSH and the web interface, as well as ipmitool from a cumin host.

The next thing that I tried was running ipmitool locally from an-worker1147.

btullis@an-worker1147:~$ sudo apt install ipmitool
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libboost-filesystem1.74.0 libboost-locale1.74.0 libboost-log1.74.0 libboost-nowide1.74.0 libboost-program-options1.74.0 libboost-thread1.74.0 libcpp-hocon0.3.0
  libfacter3.14.12 libleatherman1.12.1 libyaml-cpp0.6 linux-image-5.10.0-44-amd64 puppet python3-debconf
Use 'sudo apt autoremove' to remove them.
Recommended packages:
  openipmi
The following NEW packages will be installed:
  ipmitool
0 upgraded, 1 newly installed, 0 to remove and 88 not upgraded.
Need to get 426 kB of archives.
After this operation, 1,615 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bullseye/main amd64 ipmitool amd64 1.8.18-10.1 [426 kB]
Fetched 426 kB in 0s (17.0 MB/s)
INFO:debmonitor:Got 1 updates from dpkg hook version 3
INFO:debmonitor:Successfully sent the dpkg_hook update to the DebMonitor server(s)
Selecting previously unselected package ipmitool.
(Reading database ... 304169 files and directories currently installed.)
Preparing to unpack .../ipmitool_1.8.18-10.1_amd64.deb ...
Unpacking ipmitool (1.8.18-10.1) ...
Setting up ipmitool (1.8.18-10.1) ...
ipmievd.service is a disabled or a static unit, not starting it.
Processing triggers for man-db (2.9.4-2) ...

I can connect over the local device interface.

btullis@an-worker1147:~$ sudo ipmitool shell
ipmitool> chassis power status
Chassis Power is on

I then tried a cold reset of the bmc.

ipmitool> bmc reset cold
Sent cold reset command to MC

It was unresponsive for a few minutes, but then came back, suggesting that the cold reset worked as expected.

But I still couldn't get to the interface remotely.

So my best guess is that it's either something related to the network connection or configuration, or it's hardware failure of the network chip in the BMC itself.

If you have any ideas @VRiley-WMF, please feel free to investigate and implement whatever you feel is necessary.
I can find a workaround for the bookworm upgrade, so that we can bypass this requirement for the IPMI interface in the reimage cookbook.

This is one of our older Hadoop hosts, so we might be able to decommission it before long, but I'll check some more things before we go down that route.

BTullis mentioned this in T434494: Migrate production hadoop cluster to bookworm.Fri, Aug 14, 10:45 AM

I think the BMC has simply lost its LAN config (from impi-config --checkout):

Section Lan_Conf
        ## Possible values: Unspecified/Static/Use_DHCP/Use_BIOS/Use_Others
        IP_Address_Source                             Static
        ## Give valid IP address
        IP_Address                                    0.0.0.0
        ## Give valid MAC address
        MAC_Address                                   B0:4F:13:BB:46:5E
        ## Give valid Subnet Mask
        Subnet_Mask                                   0.0.0.0
        ## Give valid IP address
        Default_Gateway_IP_Address                    0.0.0.0
        ## Give valid MAC address
        Default_Gateway_MAC_Address                   00:00:00:00:00:00
        ## Give valid IP address
        Backup_Gateway_IP_Address                     0.0.0.0
        ## Give valid MAC address
        Backup_Gateway_MAC_Address                    00:00:00:00:00:00
EndSection

Maybe other parts (access passwords etc) also need re-commissioning.

Commencing working on this

I tried the following.

Rebooting the iDRAC - No change
Swapped the cable with 2 other cables - No change
Tried different ports on the switch - No change
Flea power drain - No change

I did look into the iDRAC settings and it's still set correctly however, the link LED indicators still are dark. This unit is out of warrenty. However, we could test and possibly use 1 of the other two ethernet ports as an iDRAC port. Switching it from "Dedicated" to "LOM3" (LOM3 and LOM4 are the other ethernet connections on the device that aren't being used, LOM1 and LOM2 are the 10G connections.) I would also put in the notes for netbox the change (as it would look different seeing it in the data center). Or we could get a replacment mainboard, but obviously we would need to purchase a replacment.

@klausman and @BTullis would either of you have an objection to trying to use LOM3 as the iDRAC connection? It doesn't seem like it would affect anything else.

VRiley-WMF mentioned this in T434926: Unresponsive management for an-worker1147.mgmt:22.Fri, Aug 14, 7:12 PM

@VRiley-WMF - Yes, please go ahead and try that other port.
I tried all sorts of things too, including sending raw IPMI commands to the BMC and manually installing a srvadmin-idracadm7 package to try various cold reset and config options from that.
Nothing has really worked so far, but one of the most interesting this has been from this.

btullis@an-worker1147:~/opt/dell/srvadmin/bin$ sudo ./idracadm7 getniccfg

IPv4 settings:
NIC Enabled          = 1
IPv4 Enabled         = 1
DHCP Enabled         = 0
IP Address           = 0.0.0.0
Subnet Mask          = 0.0.0.0
Gateway              = 0.0.0.0

IPv6 settings:
IPv6 Enabled               = Disabled
DHCP6 Enabled              = Enabled
IP Address 1               = ::
Gateway                    = ::
Link Local Address         = ::
IP Address 2               = ::
IP Address 3               = ::
IP Address 4               = ::
IP Address 5               = ::
IP Address 6               = ::
IP Address 7               = ::
IP Address 8               = ::
IP Address 9               = ::
IP Address 10              = ::
IP Address 11              = ::
IP Address 12              = ::
IP Address 13              = ::
IP Address 14              = ::
IP Address 15              = ::

LOM Status:
NIC Selection   = Dedicated
Link Detected   = No
Speed           = 10Mb/s
Duplex Mode     = Half Duplex
Active NIC      = None

Static IPv4 settings:
Static IP Address    = 10.65.1.246
Static Subnet Mask   = 255.255.255.0
Static Gateway       = 10.65.1.1

Static IPv6 settings:
Static IP Address          = ::
Static Prefix Length       = 64
Static Gateway             = ::

Notice that the static IP addresses are configured, but don't seem to be applied.

Also:

Link Detected   = No

This matches with your:

the link LED indicators still are dark.

Please have a go at using one of the other interfaces and see how we do.

Or we could get a replacment mainboard, but obviously we would need to purchase a replacment.

It's definitely not worth buying a replacement mainboard for this. The server is nearing EoL anyway, so I would say that we just try to find a workaround, for now.
If we can't even get that to work, then I will try an upgrade-in-place to get it off bullseye, then bring forward the decom date.

VRiley-WMF changed the task status from Open to In Progress.Tue, Aug 18, 12:21 PM

I made some configureations on the server and set it to LOM 3 as the iDRAC. I haven't been able to get into the iDRAC, but @BTullis said he'll continue to take a look at it.

VRiley-WMF reassigned this task from VRiley-WMF to BTullis.Tue, Aug 18, 2:33 PM

VRiley-WMF subscribed.

BTullis added a parent task: T434562: Migrate stat hosts to Bookworm or later.Wed, Aug 19, 12:49 PM

BTullis edited parent tasks, added: T434494: Migrate production hadoop cluster to bookworm; removed: T434562: Migrate stat hosts to Bookworm or later.

I have managed to get this working using the LOM3 interface.

Thanks to a tip from @VRiley-WMF, I got the netmask of the management interface correct and set it using a local idracadm7 command.

btullis@an-worker1147:~/opt/dell/srvadmin/bin$ sudo ./idracadm7 setniccfg -s 10.65.1.246 255.255.0.0 10.65.0.1
Static IP configuration enabled and modified successfully

btullis@an-worker1147:~/opt/dell/srvadmin/bin$ sudo ./idracadm7 racreset
RAC reset operation initiated successfully. It may take a few
 minutes for the RAC to come online again.

After this, I was able to ping the interface again from the cumin host and proceed with the reimage cookbook.

btullis@cumin1003:~$ ping an-worker1147.mgmt.eqiad.wmnet
PING an-worker1147.mgmt.eqiad.wmnet (10.65.1.246) 56(84) bytes of data.
64 bytes from an-worker1147.mgmt.eqiad.wmnet (10.65.1.246): icmp_seq=1 ttl=62 time=1.29 ms
64 bytes from an-worker1147.mgmt.eqiad.wmnet (10.65.1.246): icmp_seq=2 ttl=62 time=1.11 ms
^C

Created a journal entry in netbox to explain the irregular configuration of using the LOM3 interface.
https://netbox.wikimedia.org/extras/journal-entries/106/

VRiley-WMF created subtask T435390: Physical connection testing.Wed, Aug 19, 6:51 PM

VRiley-WMF closed subtask T435390: Physical connection testing as Resolved.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.Fri, Aug 21, 7:58 AM

Unresponsive management for an-worker1147.mgmt:22Closed, ResolvedPublicActions