Page MenuHomePhabricator
Create Task
Maniphest T312298

Update NIC firmware on all Elastic PowerEdge R440 elastic hosts
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Before we can update our hosts to Bullseye, we'll need to update the firmware on Dell PowerEdge R440 with Broadcom NetExtreme NICs (the installer won't work until the NIC firmware is updated). As of this writing, the current firmware version is 21.85.21.92 .

AC:

  • All hosts updated to latest approved NIC firmware
  • Process is thoroughly tested and documented.

See https://phabricator.wikimedia.org/T309343#7971329 and https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/Dell_Documentation#Urgent_Firmware_Revision_Notices: for more details.

Related Objects
Search...

Event Timeline

bking created this task.Jul 6 2022, 9:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 6 2022, 9:08 PM
bking added a parent task: T309343: Upgrade cloudelastic clusters to Debian Bullseye.Jul 6 2022, 9:08 PM
bking mentioned this in P30946 downtime cookbook.Jul 7 2022, 1:19 PM
bking renamed this task from Update firmware on all elastic hosts to Update NIC firmware on all Elastic PowerEdge R440 elastic hosts.Jul 11 2022, 4:22 PM
bking closed this task as Invalid.
bking updated the task description. (Show Details)
bking updated the task description. (Show Details)
bking added a parent task: T289135: Upgrade Cirrus Elasticsearch clusters to Debian Bullseye.Jul 11 2022, 4:26 PM
bking added a comment.Jul 11 2022, 4:44 PM

Looks like Puppetboard has the relevant information, here is an example entry for a single host.

Using this info, I made a list of all the Elastic hosts with R440 Chassis .

The NIC firmware version info is in PuppetDB too, but it looks like it'll require a more complex query. Working on it...

bking reopened this task as In Progress.Jul 11 2022, 9:05 PM
bking edited projects, added Discovery-Search (Current work); removed Discovery-Search.Jul 11 2022, 9:49 PM
Stashbot added a comment.Jul 12 2022, 3:48 PM

Mentioned in SAL (#wikimedia-operations) [2022-07-12T15:48:48Z] <bking@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic1065.eqiad.wmnet with reason: firmware update T312298

Stashbot added a comment.Jul 12 2022, 3:49 PM

Mentioned in SAL (#wikimedia-operations) [2022-07-12T15:49:01Z] <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic1065.eqiad.wmnet with reason: firmware update T312298

Stashbot added a comment.Jul 12 2022, 7:19 PM

Mentioned in SAL (#wikimedia-operations) [2022-07-12T19:19:53Z] <bking@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on elastic2038.codfw.wmnet with reason: firmware update T312298

Stashbot added a comment.Jul 12 2022, 7:20 PM

Mentioned in SAL (#wikimedia-operations) [2022-07-12T19:20:07Z] <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on elastic2038.codfw.wmnet with reason: firmware update T312298

bking added a comment.Jul 14 2022, 3:22 PM

Firmware is staged on all R440 hosts, will be automatically applied during the reimage. I've reimaged several hosts already; I can verify that the firmware updates apply correctly without disrupting the reimage process.

bking mentioned this in T313052: Reimage non-production Elastic hosts to Bullseye.Jul 14 2022, 3:51 PM
MPhamWMF set the point value for this task to 3.Jul 18 2022, 3:56 PM
MPhamWMF moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.
bking closed this task as Resolved.Aug 3 2022, 8:22 PM

After some though, I think the current state (completely applied in CODFW, to be finished in EQIAD) is enough to close. We've had 100% success in CODFW and I don't foresee a problem in EQIAD .

Follow the EQIAD work in T289135 .

bking mentioned this in T331297: Audit/update NIC firmware on Search Platform-owned Buster hosts.Mar 6 2023, 2:58 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL