@RobH - can you work with the traffic to get the bios upgraded on the cp hosts in esams? In T240177, @Papaul
found a Dell bulletin that associated with the random crashes:

"Fixed a continuous reboot issue and Out of Resource error with PCIe IO resource allocation which was observed in the 2.4.7 version." see link below:
https://www.dell.com/support/home/us/en/19/drivers/driversdetails?driverid=wgm2r&oscode=wst14&productcode=poweredge-r440"

Thanks,
Willy

@BBlack, Can we modify this task to include the eqiad caches that need update as well? I'll be handing these remotely. During this process, if any single server fails and requires on-site work, I'll make a sub-task for its repair off this task.

If that is acceptable, we'll just need to add the eqiad cp systems to be updated into the task description. As I understand it now, we want to update ALL R440 bios versions in our cp clusters, correct?

@RobH - Yes, let's edit this to include eqiad as well. We've had the same symptoms both places, and they're the same approximate generation of hardware configuration (IIRC, only the NVMe changed to a slightly newer/better model from eqiad to esams, but the base system is otherwise fairly identical).

As for process - in general we should be able to cleanly shut these down from the OS level, and they'll self-depool during their shutdown. When a node is ready to go back into service, the command pool from a root prompt will re-pool its services into production use. As long as icinga checks for the nodes are coming back all-green after the reboot (some may not until the initial puppet run has completed!), we should normally be good to repool. Only other caveat is watch out for the ongoing Debian Buster upgrades as well - if it's EU time (US morning), check with Traffic channel in case they're upgrading the same nodes.

My plan is to do one from each service group (upload/text) at a time, batched together. (It is just as easy to watch two bios updates as one, it doesn't quite scale more than that for close supervision.) Would that be ok?

Mentioned in SAL (#wikimedia-operations) [2020-02-10T21:38:30Z] <robh> cp1075 & cp1076 offline for bios updates per T243167

Mentioned in SAL (#wikimedia-operations) [2020-02-10T21:58:41Z] <robh> cp107[56] returned to service, cp107[78] offline for bios update via T243167

Mentioned in SAL (#wikimedia-operations) [2020-02-10T22:42:10Z] <robh> cp107[89] returned to service, cp108[01] offline for bios update via T243167

Mentioned in SAL (#wikimedia-operations) [2020-02-10T22:43:23Z] <robh> cp107[789] returned to service, cp108[01] offline for bios update via T243167

Mentioned in SAL (#wikimedia-operations) [2020-02-10T23:06:05Z] <robh> cp108[01] returned to service, cp108[23] offline for bios update via T243167

Mentioned in SAL (#wikimedia-operations) [2020-02-10T23:30:14Z] <robh> cp108[23] returned to service via T243167

Mentioned in SAL (#wikimedia-operations) [2020-02-11T20:38:12Z] <robh> depooling cp108[45] for firmware update via T243167

Mentioned in SAL (#wikimedia-operations) [2020-02-11T20:57:30Z] <robh> cp108[45] returned to service, depooling cp108[67]for firmware update via T243167

Please note the outage caused the SAL of my adding back cp108[67] to service, (as the rest weren't really returned, but depooled.)

Mentioned in SAL (#wikimedia-operations) [2020-02-19T17:12:36Z] <robh> cp1088 returned to service, cp1089 & cp1090 offline for firmware update via T243167

Mentioned in SAL (#wikimedia-operations) [2020-02-19T17:48:16Z] <robh> cp1089 cp1090 returned to service via T243167

Please note as of now, all eqiad cp sysetms have been updated to the latest bios revision. If these hosts experience any further crashes, its vital to note it for review.

I'll move onto the esams cp systems in my afternoon, as that will be closer to post-peak esams time.

Mentioned in SAL (#wikimedia-operations) [2020-02-19T22:37:52Z] <robh> taking cp3050 & cp3051 offline for firmware update via T243167

Mentioned in SAL (#wikimedia-operations) [2020-02-19T22:54:57Z] <robh> cp3050 & cp3051 returned to service via T243167

This has sat ignored while I was doing procurement, but I'll pick this back up next week and start updating these again. (On clinic duty this week so I don't want to split my attention quite that much.)

So there is currently an experiment going on with caching hosts in esams, and flashing firmware would interrupt that. T264398#6772586 When that is done, this can resume, likely in a week or so.

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

Can we resurrect this and I finish out the esams hosts? I'd like to close this out, its just shaming me with its age.

Checking first since the traffic team could be running testing which would be affected by a rolling restart of the majority of esams hosts (in a regular cadence, but it would affect test metrics.)

Synced with Brandon via IRC, and I'm good to resume this. Each host, one per cluster at a time (one upload, one text), disabling puppet agent, depooling, rebooting for firmware, then repooling and reenabling puppet.

Mentioned in SAL (#wikimedia-sre) [2022-05-11T21:01:10Z] <robh> cp305[23] going offline via T243167 for firmware updates (puppet agent disabled and depooled prior to reboot)

Mentioned in SAL (#wikimedia-sre) [2022-05-11T21:34:00Z] <robh> cp50[23] returned to service and all green in icinga, cp50[45] depooling for firmware update T243167

Mentioned in SAL (#wikimedia-sre) [2022-05-11T21:34:37Z] <robh> cp30[23] returned to service and all green in icinga, cp30[45] depooling for firmware update T243167

Mentioned in SAL (#wikimedia-sre) [2022-05-11T22:00:55Z] <robh> cp305[45] returned to service and all green in icinga, cp305[67] depooling for firmware update T243167

Mentioned in SAL (#wikimedia-sre) [2022-05-11T22:28:12Z] <robh> cp305[67] returned to service and all green in icinga, cp305[89] depooling for firmware update T243167

Mentioned in SAL (#wikimedia-sre) [2022-05-12T20:50:30Z] <robh> resuming last 6 esams cp host firmware updates via T243167. cp306[01] going offline

cp3060 refuses to load its idrac https interface, even when i clear browser history and do a racreset on the idrac interface, skipping it and continuing with the rest of them.

Mentioned in SAL (#wikimedia-sre) [2022-05-12T21:15:22Z] <robh> cp306[01] returned to service, cp306[23] coming down for firmware update via T243167

Mentioned in SAL (#wikimedia-sre) [2022-05-12T21:43:12Z] <robh> cp306[23] returned to service, cp306[45] coming down for firmware update via T243167

Mentioned in SAL (#wikimedia-operations) [2022-05-19T22:07:30Z] <robh> cp3060 idrac interface frozen, rebooted via power outlet control on T243167

Mentioned in SAL (#wikimedia-operations) [2022-05-20T08:53:58Z] <vgutierrez> re-enabling puppet and repooling cp3060 - T308797 T243167