Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | ayounsi | T396712 Evaluate automatic MAC-based DHCP for production servers | |||
| Resolved | Jhancock.wm | T396718 cirrussearch2079 iDRAC not working |
Event Timeline
double checking before i do something. Is this server depooled/drained? an idrac firmware upgrade does require a reboot.
I added the Data-Platform-SRE tag to the task, I think @bking was recently working on those hosts.
Looks like it's still failing, an upgrade is probably still needed.
>>> r =spicerack.redfish('cirrussearch2079')
Management Password:
>>> r.get_primary_mac()
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 489, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 884, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 884, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 884, in urlopen
return self.urlopen(
^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 874, in urlopen
retries = retries.increment(method, url, response=response, _pool=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 594, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.193.1.6', port=443): Max retries exceeded with url: /redfish/v1/Systems/System.Embedded.1 (Caused by ResponseError('too many 500 error responses'))@BTullis can you depool this server for me so i can do the upgrade. It will require a reboot (or two) of the server.
@bking i rebooted the idrac on this server. It's about all i can do for the moment. is there a time we can depool this server so i can try to upgrade the idrac firmware?
Mentioned in SAL (#wikimedia-operations) [2025-07-24T14:24:38Z] <bking@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cirrussearch2079.codfw.wmnet with reason: T396718
@Jhancock.wm Sorry we did not respond to this one sooner. The host is downtimed and depooled, feel free to reboot whenever.
updated the idrac firmware manually. then tested a run of firmware update script to see if it would connect. It does. So the problem should be fixed. and the bios firmware is updated now as well. please confirm and then we can close. ty!