While investigating T269313 I checked the firmware versions of all the cloudvirts. On Dell systems we seem to be standardized on two versions, /except/ for cloudvirt1023 which seems to have an ancient, different version.
Let's try to understand that and possibly upgrade.
1023 was purchased at the same time as 1024. Here is 1024:
node_nic_firmware_version{cluster="wmcs", device="ens3f0np0", driver="bnxt_en", firmware_version="214.4.32.0/pkg 21.60.22.11", instance="cloudvirt1024:9100", job="node", site="eqiad"}here is 1023:
node_nic_firmware_version{cluster="wmcs", device="ens3f0np0", driver="bnxt_en", firmware_version="20.6.151.0", instance="cloudvirt1023:9100", job="node", site="eqiad"}Mentioned in SAL (#wikimedia-cloud) [2020-12-04T22:23:59Z] <andrewbogott> moving cloudvirt1023 out of the ceph aggregate and into maintenance for T269467
It turns out this has a weird firmware version because it's weird hardware
robh: this is a qlogic 41112 sfp adapter which is not quite the same as others
Given that it isn't broken, let's leave it as it is.
Mentioned in SAL (#wikimedia-cloud) [2020-12-04T22:33:54Z] <andrewbogott> moving cloudvirt1023 back into the ceph aggregate; it doesn't need upgrades after all T269467
It was indeed ordered originally with odd HW, the QLogic 41112, but we replaced those a long time ago with standard Broadcom 57412s (T203827). So these actually have the same NICs (57412) as the other cloudvirts - hence reopening!
Mentioned in SAL (#wikimedia-cloud) [2020-12-05T00:35:35Z] <andrewbogott> moving cloudvirt1023 back into maintenance because T269467 continues to puzzle
Mentioned in SAL (#wikimedia-cloud) [2020-12-07T18:33:50Z] <andrewbogott> putting cloudvirt1023 back into service T269467
confirmed:
root@cloudvirt1023:~# ethtool -i ens3f1np1 | grep firmware firmware-version: 214.4.91.1/pkg 21.65.33.33
I've put this host back in service.