After upgrading db1104's kernel (this is s8 eqiad master) the host didn't come back up.
This is probably another case of T216240, we need to get a maintenance window with @Cmjohnson to get its firmware upgraded.
For now I have just started it with an older kernel
| Project | Branch | Lines +/- | Subject | |
|---|---|---|---|---|
| operations/puppet | production | +0 -1 | db1104: Enable notifications |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Declined | None | T216240 Reboot, upgrade firmware and kernel of db1096-db1106, db2071-db2092 | |||
| Resolved | Jclark-ctr | T286226 Upgrade db1104 firmware |
@wiki_willy we're good to move ahead with this one. The host needs to be downtimed and shut down in advance, what (EU friendly) time would work for you?
I can handle this, and a firmware upgrade takes anywhere from 5 to 30 minutes (depending on if its only bios, etc..) This is just bios, so I can handle this anytime this week.
@LSobanski: Please have someone take it offline and then assign this task back to me for the actual firmware update. I can handle this anytime this week, no problem!
Firmware Update Notes:
Workflow:
Mentioned in SAL (#wikimedia-operations) [2021-07-13T16:57:59Z] <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1104.eqiad.wmnet with reason: Firmware upgrade T286226
Mentioned in SAL (#wikimedia-operations) [2021-07-13T16:58:05Z] <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1104.eqiad.wmnet with reason: Firmware upgrade T286226
Mentioned in SAL (#wikimedia-operations) [2021-07-13T16:59:20Z] <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on 18 hosts with reason: Firmware upgrade T286226
Mentioned in SAL (#wikimedia-operations) [2021-07-13T16:59:27Z] <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 18 hosts with reason: Firmware upgrade T286226
So for best practices, I tend to upload a new idrac firmware before anything else. Since idrac handles the firmware updates of other things, it just seems safest.
That being stated, this host is taking forever to update the idrac firmware properly. I started it immediately after Kormat depooled the host, and its now in idrac reboot (host accessible, idrac down) as I wait for it to come back so I can update the bios.
This has gone exceedingly poorly.
I updated the idrac firmware, and idrac is accessible via SSH but not via HTTPS, as its self signed cert changed to one unsupported (even with bypass option via advanced button) in browser.
I think the solution then is to crashcart and apply the bios update via usb stick or roll the idrac version back one to a version that doesn't break idrac. Without a crash cart connection though, I'm not sure how to roll this back to an earlier version.
I'm hunting around the internet for potential solutions that dont involve our onsites laying hands on the host.
I am at a loss on how to proceed for this, without connecting a crash cart and rolling back the idrac version via crash cart. Basically the latest idrac firmware made the https idrac interface unreachable. That interface is what we use to flash the bios firmware, so now we're stuck with outdated bios on this host.
@LSobanski: We won't have our normal on-sites there until next week. This host is unfortunately outside of support coverage from Dell, so we cannot have them dispatch a tech. We will need to fix this ourselves, and I'm not 100% certain of the process of using the crash cart to access and roll back firmware, so it would be blind leading the blind if I tried to walk a EQ tech through it.
Thoughts?
@RobH Thanks for digging into it. Let's wait until we have people onsite and can crash cart into the host. Who would be the best person to assign this to so it gets scheduled as soon as possible?
FYI: My understanding is with a crash cart, one can activate the lifecycle controller manually on rebooting the host in POST and then fail backwards to the last version of any firmware pushed. This option is available via the HTTPS interface, which is not reachable due to this issue.
It looks like Chris is going to be out for a while. Moving this task over to @Jclark-ctr, who should be back Tuesday or Wednesday. Thanks, Willy
That's around 10pm for me (and for @Kormat) and 9pm for @LSobanski, any chances this can be done a bit earlier? Another option would be to leave the host down for you to work it out and power it back on once you are done.
I would prefer leave host down for me to work it out and power it back when finished unless @Cmjohnson is available earlier
@Kormat I can update f/w today, please take the server offline and I run the f/w updates.
Mentioned in SAL (#wikimedia-operations) [2021-08-04T14:13:28Z] <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 18 hosts with reason: Firmware upgrade on db1104 (s8 primary) T286226
Mentioned in SAL (#wikimedia-operations) [2021-08-04T14:13:42Z] <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 18 hosts with reason: Firmware upgrade on db1104 (s8 primary) T286226
@Kormat The f/w update has been completed. I am able to ssh into the host. resolving the task, if you have continue to have issues please let me know.
Change 710729 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1104: Enable notifications
Change 710729 merged by Marostegui:
[operations/puppet@production] db1104: Enable notifications