Hi Chris,
I know that you were missing this but one disk just broke on analytics1055. This is a regular Hadoop node, already disabled puppet and all the related daemons (+ downtime):
[Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 01 d1 af ff f0 00 00 00 08 00 00 [Tue Aug 8 18:30:40 2017] blk_update_request: I/O error, dev sdd, sector 7812939760 [Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [Tue Aug 8 18:30:40 2017] blk_update_request: I/O error, dev sdd, sector 0 [Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 08 00 00 00 08 00 00 [Tue Aug 8 18:30:40 2017] blk_update_request: I/O error, dev sdd, sector 8 [Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00 [Tue Aug 8 18:30:40 2017] blk_update_request: I/O error, dev sdd, sector 0 [Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 01 d1 af f7 f0 00 00 00 08 00 00 [Tue Aug 8 18:30:40 2017] blk_update_request: I/O error, dev sdd, sector 7812937712 [Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 08 00 00 00 00 08 00 00 [Tue Aug 8 18:30:40 2017] blk_update_request: I/O error, dev sdd, sector 2048 [Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 08 08 00 00 00 08 00 00 [Tue Aug 8 18:30:40 2017] blk_update_request: I/O error, dev sdd, sector 2056 [Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [Tue Aug 8 18:30:40 2017] sd 0:2:2:0: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 08 00 00 00 00 08 00 00 [Tue Aug 8 18:30:40 2017] blk_update_request: I/O error, dev sdd, sector 2048 [Tue Aug 8 18:30:50 2017] scsi_io_completion: 22 callbacks suppressed [Tue Aug 8 18:30:50 2017] sd 0:2:2:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [Tue Aug 8 18:30:50 2017] sd 0:2:2:0: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 01 8b 81 c1 48 00 00 00 08 00 00 [Tue Aug 8 18:30:50 2017] blk_update_request: 22 callbacks suppressed [Tue Aug 8 18:30:50 2017] blk_update_request: I/O error, dev sdd, sector 6635503944 [Tue Aug 8 18:30:50 2017] sd 0:2:2:0: [sdd] tag#1 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [Tue Aug 8 18:30:50 2017] sd 0:2:2:0: [sdd] tag#1 CDB: Read(16) 88 00 00 00 00 01 8b 81 b7 d8 00 00 00 08 00 00 [Tue Aug 8 18:30:50 2017] blk_update_request: I/O error, dev sdd, sector 6635501528 [Tue Aug 8 18:30:50 2017] EXT4-fs error (device sdd1): ext4_find_entry:1463: inode #207356828: comm java: reading directory lblock 0 [Tue Aug 8 18:30:50 2017] EXT4-fs (sdd1): previous I/O error to superblock detected [Tue Aug 8 18:30:50 2017] sd 0:2:2:0: [sdd] tag#1 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [Tue Aug 8 18:30:50 2017] sd 0:2:2:0: [sdd] tag#1 CDB: Write(16) 8a 00 00 00 00 00 00 00 08 00 00 00 00 08 00 00 [Tue Aug 8 18:30:50 2017] blk_update_request: I/O error, dev sdd, sector 2048 [Tue Aug 8 18:30:50 2017] Buffer I/O error on dev sdd1, logical block 0, lost sync page write [Tue Aug 8 18:30:50 2017] EXT4-fs error (device sdd1): ext4_find_entry:1463: inode #207356850: comm java: reading directory lblock 0 [Tue Aug 8 18:30:50 2017] EXT4-fs error (device sdd1): ext4_find_entry:1463: inode #207356850: comm java: reading directory lblock 0 [Tue Aug 8 18:30:50 2017] EXT4-fs (sdd1): previous I/O error to superblock detected [Tue Aug 8 18:30:50 2017] sd 0:2:2:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [Tue Aug 8 18:30:50 2017] sd 0:2:2:0: [sdd] tag#0 CDB: Write(16) 8a 00 00 00 00 00 00 00 08 00 00 00 00 08 00 00 [Tue Aug 8 18:30:50 2017] blk_update_request: I/O error, dev sdd, sector 2048 [Tue Aug 8 18:30:50 2017] Buffer I/O error on dev sdd1, logical block 0, lost sync page write [Tue Aug 8 18:30:50 2017] EXT4-fs (sdd1): previous I/O error to superblock detected [Tue Aug 8 18:30:50 2017] sd 0:2:2:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [Tue Aug 8 18:30:50 2017] sd 0:2:2:0: [sdd] tag#0 CDB: Write(16) 8a 00 00 00 00 00 00 00 08 00 00 00 00 08 00 00 [Tue Aug 8 18:30:50 2017] blk_update_request: I/O error, dev sdd, sector 2048 [Tue Aug 8 18:30:50 2017] Buffer I/O error on dev sdd1, logical block 0, lost sync page write
Virtual Drive: 2 (Target Id: 2) Name : RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0 Size : 3.637 TB Sector Size : 512 Is VD emulated : No Parity Size : 0 State : Offline <<<<======================= Strip Size : 64 KB Number Of Drives : 1 Span Depth : 1 Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU Current Cache Policy: WriteThrough, ReadAheadNone, Direct, No Write Cache if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Disk's Default Preserved Cache Data: Yes Encryption Type : None Default Power Savings Policy: Controller Defined Current Power Savings Policy: None Can spin up in 1 minute: Yes LD has drives that support T10 power conditions: Yes LD's IO profile supports MAX power savings with cached writes: No Bad Blocks Exist: No PI type: No PI
Could you please replace it whenever you have time?
Thanks!
Luca