db1082 paged for replication lag, which was actually a replication being stopped.
The reason for it was that MySQL crashed
170215 13:56:41 [ERROR] InnoDB: Tried to read 16384 bytes at offset 28488318976. Was only able to read 0. 2017-02-15 13:56:41 7fdf53d9d700 InnoDB: Operating system error number 5 in a file operation. InnoDB: Error number 5 means 'Input/output error'. InnoDB: Some operating system error numbers are described at InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html 170215 13:56:41 [ERROR] InnoDB: File (unknown): 'read' returned OS error 105. Cannot continue operation 170215 13:56:56 mysqld_safe Number of processes running now: 0 170215 13:56:56 mysqld_safe mysqld restarted
That happens a bit after this:
Feb 15 13:56:41 db1082 kernel: [11085714.884288] sd 0:1:0:0: [sda] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Feb 15 13:56:41 db1082 kernel: [11085714.884426] sd 0:1:0:0: [sda] tag#0 Sense Key : Aborted Command [current] Feb 15 13:56:41 db1082 kernel: [11085714.884571] sd 0:1:0:0: [sda] tag#0 Add. Sense: Information unit iuCRC error detected Feb 15 13:56:41 db1082 kernel: [11085714.884574] sd 0:1:0:0: [sda] tag#0 CDB: Read(16) 88 00 00 00 00 01 13 cd da a0 00 00 00 20 00 00 Feb 15 13:56:41 db1082 kernel: [11085714.884722] blk_update_request: I/O error, dev sda, sector 4627225248
Which looks storage related.
There are no logs on the ILO
Server is depooled
The plan is to:
- Reboot the server
- Check if everything is ok and/or logs are generated