Connecting to the BMC as user root (wmf_root_mgmt)
Testing Redfish API connection to cloudvirt1077 (10.65.1.144)
The response payload does not contain any valid JSON to log.
Failed to run cookbooks.sre.hosts.provision.SupermicroProvisionRunner._try_bmc_password: Client Response error when trying to contact the BMC.
==> What do you want to do? "retry" the last command, manually fix the issue and "skip" the last command to continue the execution or completely "abort" the execution.
>

elukey@install1005:~$ sudo journalctl -u isc-dhcp-server.service --since '2 hours ago' | grep 10.65.1.147
May 04 15:16:44 install1005 dhcpd[390969]: DHCPOFFER on 10.65.1.147 to 90:5a:08:29:67:25 via 10.65.0.1
May 04 15:16:44 install1005 dhcpd[390969]: DHCPREQUEST for 10.65.1.147 (208.80.154.134) from 90:5a:08:29:67:25 via 10.65.0.1
May 04 15:16:44 install1005 dhcpd[390969]: DHCPACK on 10.65.1.147 to 90:5a:08:29:67:25 via 10.65.0.1

Running IPMI command: ipmitool -I lanplus -H cloudvirt1078.mgmt.eqiad.wmnet -U root -E chassis power status
Error: Unable to establish IPMI v2 / RMCP+ session
Exception raised while executing cookbook sre.hosts.provision:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/ipmi.py", line 90, in command
    output = run(command + command_parts, env=self.env.copy(), stdout=PIPE, check=True).stdout.decode()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ipmitool', '-I', 'lanplus', '-H', 'cloudvirt1078.mgmt.eqiad.wmnet', '-U', 'root', '-E', 'chassis', 'power', 'status']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 265, in _run
    raw_ret = runner.run()
              ^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/hosts/provision.py", line 415, in run
    self.ipmi.check_connection()
  File "/usr/lib/python3/dist-packages/spicerack/ipmi.py", line 105, in check_connection
    self.power_status()
  File "/usr/lib/python3/dist-packages/spicerack/ipmi.py", line 115, in power_status
    status = self.command(["chassis", "power", "status"], is_safe=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/spicerack/ipmi.py", line 92, in command
    raise IpmiError(f"Remote IPMI for {self._target} failed (exit={e.returncode}): {e.output}") from e
spicerack.ipmi.IpmiError: Remote IPMI for cloudvirt1078.mgmt.eqiad.wmnet failed (exit=1): b''
Released lock for key /spicerack/locks/cookbooks/sre.hosts.provision:cloudvirt1078: {'concurrency': 1, 'created': '2026-05-04 17:31:38.217788', 'owner': 'jclark@cumin1003 [4071054]', 'ttl': 1800}
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART

Updating the root user's password on the BMC.
Changing password for the account with username root: /redfish/v1/AccountService/Accounts/3
Updating the ADMIN user's password on the BMC.
Changing password for the account with username ADMIN: /redfish/v1/AccountService/Accounts/2
Updated current instance password to the new password
Running IPMI command: ipmitool -I lanplus -H cloudvirt1077.mgmt.eqiad.wmnet -U root -E chassis power status
Error: Unable to establish IPMI v2 / RMCP+ session

cloudvirt1078 (WMF11006): unsupported case, found 2 v4 prefixes and 1 v6 prefixes, expected 1 and 0 or 1 respectively, skipping IP allocation.

[info] Called with parameters: {'device': <Device: cloudvirt1078 (WMF11006)>, 'z_port': 37, 'interface_type': '10gbase-x-sfpp', 'cable_id': '', 'z_nbdevice': None, 'vlan_type': 'cloud-hosts', 'vlan': None, 'mgmt_mac': '90:5a:08:29:6b:26'}
[warning] cloudvirt1078 (WMF11006): Skipping assignment of MGMT interface because already allocated
[success] cloudvirt1078 (WMF11006): created interface ##PRIMARY## (mgmt=False)
[debug] cloudvirt1078 (WMF11006): selecting address from prefix 10.64.150.0/24
[success] cloudvirt1078 (WMF11006): assigned IPv4 10.64.150.2/24 to interface ##PRIMARY## with DNS name 'cloudvirt1078.eqiad.wmnet'.
[success] cloudvirt1078 (WMF11006): marked IPv4 address 10.64.150.2/24 as primary IPv4 for device.
[success] cloudvirt1078 (WMF11006): assigned IPv6 2620:0:861:11e:10:64:150:2/64 to interface ##PRIMARY## with DNS name 'cloudvirt1078.eqiad.wmnet'.
[success] cloudvirt1078 (WMF11006): marked IPv6 address 2620:0:861:11e:10:64:150:2/64 as primary IPv6.
[success] cloudsw1-d5-eqiad (WMF4808):xe-0/0/37 configured vlan.
[warning] cloudvirt1078 (WMF11006): assuming Black  because 10gbase-x-sfpp
[success] cloudvirt1078 (WMF11006): created cable #11250