ms-be1022 had hung. Powercycling it and looking at iLO log showed the following:
313-HPE Smart Storage Battery 1 Failure - Battery Shutdown Event Code: 0x0400. Action: Restart system. Contact HPE support if condition persists.
</>hpiLO-> show /system1/log1/record41
status=0
status_tag=COMMAND COMPLETED
Sun Nov 15 13:42:00 2020
/system1/log1/record41
Targets
Properties
number=41
severity=Caution
date=11/15/2020
time=13:15
description=Smart Storage Battery failure (Battery 1, service information: 0x0A). Action: Gather AHS log and contact Support
Verbs
cd version exit showWe're also getting I/O errors on disk sdb, which is one of the mirrors for the root filesystem:
Nov 15 14:02:41 ms-be1022 kernel: [ 11.586463] sd 0:1:0:1: [sdb] tag#13 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Nov 15 14:02:41 ms-be1022 kernel: [ 11.586468] sd 0:1:0:1: [sdb] tag#13 Sense Key : Aborted Command [current] Nov 15 14:02:41 ms-be1022 kernel: [ 11.586473] sd 0:1:0:1: [sdb] tag#13 Add. Sense: Information unit iuCRC error detected Nov 15 14:02:41 ms-be1022 kernel: [ 11.586478] sd 0:1:0:1: [sdb] tag#13 CDB: Read(10) 28 00 17 48 db f0 00 00 08 00 Nov 15 14:02:41 ms-be1022 kernel: [ 11.586481] blk_update_request: I/O error, dev sdb, sector 390650864