It looks like the errors / timeouts on newer ms-be machines might be due to a missing firmware upgrade for the hardware raid controller P840. We can test an upgrade on one of the machines in codfw to begin with though and expand to other hp machines that would need the upgrade, possibly other controller models too.
codfw ms-be hp firware:
```lines=5
root@neodymium:~# salt -C 'G@manufacturer:hp and G@cluster:swift and G@site:codfw' cmd.run 'hpssacli controller slot=3 show | grep -i firmware'
ms-be2018.codfw.wmnet:
Firmware Version: 3.00
ms-be2021.codfw.wmnet:
Firmware Version: 3.00
ms-be2016.codfw.wmnet:
Firmware Version: 3.00
ms-be2017.codfw.wmnet:
Firmware Version: 3.00
ms-be2019.codfw.wmnet:
Firmware Version: 3.00
ms-be2020.codfw.wmnet:
Firmware Version: 3.00
ms-be2027.codfw.wmnet:
Firmware Version: 3.56
ms-be2024.codfw.wmnet:
Firmware Version: 3.56
ms-be2026.codfw.wmnet:
Firmware Version: 3.56
ms-be2023.codfw.wmnet:
Firmware Version: 3.56
ms-be2025.codfw.wmnet:
Firmware Version: 3.56
ms-be2022.codfw.wmnet:
Firmware Version: 3.56
```
eqiad:
```lines=4
root@neodymium:~# salt -C 'G@manufacturer:hp and G@cluster:swift and G@site:eqiad' cmd.run 'hpssacli controller slot=3 show | grep -i firmware'
ms-be1017.eqiad.wmnet:
ms-be1018.eqiad.wmnet:
ms-be1016.eqiad.wmnet:
ms-be1019.eqiad.wmnet:
Firmware Version: 3.00
ms-be1022.eqiad.wmnet:
Firmware Version: 3.56
ms-be1020.eqiad.wmnet:
Firmware Version: 3.00
ms-be1021.eqiad.wmnet:
Firmware Version: 3.00
ms-be1025.eqiad.wmnet:
Firmware Version: 3.56
ms-be1024.eqiad.wmnet:
Firmware Version: 3.56
ms-be1026.eqiad.wmnet:
Firmware Version: 3.56
ms-be1023.eqiad.wmnet:
Firmware Version: 3.56
# older machines have controller on slot=1
root@neodymium:~# salt -C 'G@manufacturer:hp and G@cluster:swift and G@site:eqiad' cmd.run 'hpssacli controller slot=1 show | grep -i firmware'
ms-be1021.eqiad.wmnet:
ms-be1022.eqiad.wmnet:
ms-be1019.eqiad.wmnet:
ms-be1020.eqiad.wmnet:
ms-be1025.eqiad.wmnet:
ms-be1023.eqiad.wmnet:
ms-be1026.eqiad.wmnet:
ms-be1024.eqiad.wmnet:
ms-be1018.eqiad.wmnet:
Firmware Version: 1.34
ms-be1017.eqiad.wmnet:
Firmware Version: 1.34
ms-be1016.eqiad.wmnet:
Firmware Version: 1.34
```
= Links to firmware downloads =
* [[ https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_2729501f721b42dab91f5cfe80 | Smart Array H240ar, H240nr, H240, H241, H244br, P240nr, P244br, P246br, P440ar, P440, P441, P542D, P741m, P840, P840ar, and P841 ]] version 6.30(5 Feb 2018)
* [[ https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_42b6aa58956a438aa85bd73d0f | Smart Array P220i, P222, P420i, P420, P421, P721m, and P822 ]] version 8.32(2 Nov 2017)