> Date/Time: Tue Sept 6 16:30:43 UTC 2016
Icinga has registered a service failure of CQL on restbase2004-b.codfw.wmnet.
There are 69 logged `org.apache.cassandra.io.sstable.CorruptSSTableException` exceptions start, the result of a shutdown after encountering at 2016-09-06 16:24:37,145data corruption.
```
ERROR [CompactionExecutor:12945] 2016-09-06 16:24:37,145 CassandraDaemon.java:185 - Exception in threa
d Thread[CompactionExecutor:12945,1,main]
org.apache.cassandra.io.FSReadError: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-35707-big-Data.db
at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:358) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:359) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:322) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:126) ~[apache-cassandra-2.2.6.jar:2.2.6]
...[...]
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-35707-big-Data.db
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferStandard(CompressedRandomAccessReader.java:153) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:230) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.io.compress.CompressedThrottledReader.reBuffer(CompressedThrottledReader.java:42) ~[apache-cassandra-2.2.6.jar:2.2.6]
at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:346) ~[apache-cassandra-2.2.6.jar:2.2.6]
... 30 common frames omitted
Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-35707-big-Data.db): corruption detected, chunk at 24958152 of length 57619.
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferStandard(CompressedRandomAccessReader.java:124) ~[apache-cassandra-2.2.6.jar:2.2.6]
... 33 common frames omitted
ERROR [CompactionExecutor:12945] 2016-09-06 16:24:37,147 StorageService.java:467 - Stopping gossiper
WARN [CompactionExecutor:12945] 2016-09-06 16:24:37,147 StorageService.java:373 - Stopping gossip by operator request
INFO [CompactionExecutor:12945] 2016-09-06 16:24:37,147 Gossiper.java:1448 - Announcing shutdown
INFO [CompactionExecutor:12945] 2016-09-06 16:24:37,149 StorageService.java:1937 - Node /10.192.32.138 state jump to shutdown
ERROR [CompactionExecutor:12945] 2016-09-06 16:24:39,170 StorageService.java:477 - Stopping native transport
INFO [CompactionExecutor:12945] 2016-09-06 16:24:39,255 Server.java:218 - Stop listening for CQL clients
```
NOTE: All are for `/srv/cassandra-b/data/local_group_wikipedia_T_parsoid_html/data-f3648bc0c2cb11e49ce6a1da77f2fd34/la-35707-big-Data.db`
AndAdditionally, this was found in dmesg:
```
10923014.250916] hpsa 0000:03:00.0: scsi 0:1:0:1 Aborting command ffff880cd6f1c9c0Tag:0x00000000:00000120 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT
[10923014.250922] hpsa 0000:03:00.0: scsi 0:1:0:1: Aborting command Direct-Access HP LOGICAL VOLUME RAID-0 SSDSmartPathCap+ En+ Exp=1
[10923014.250960] hpsa 0000:03:00.0: scsi 0:1:0:1 Aborting command ffff880cd6f1c9c0Tag:0x00000000:00000120 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, FAILED
[10923014.250968] hpsa 0000:03:00.0: scsi 0:1:0:1: FAILED to abort command Direct-Access HP LOGICAL VOLUME RAID-0 SSDSmartPathCap+ En+ Exp=1
[10923031.235036] hpsa 0000:03:00.0: scsi 0:1:0:1: resetting logical Direct-Access HP LOGICAL VOLUME RAID-0 SSDSmartPathCap+ En+ Exp=1
[10923046.114183] hpsa 0000:03:00.0: aborted: LUN:000000c000000101 CDB:12000000310000000000000000000000
[10923046.114189] hpsa 0000:03:00.0: hpsa_update_device_info: inquiry failed
[10923046.146611] hpsa 0000:03:00.0: Inquiry failed, skipping device.
[10923046.163938] hpsa 0000:03:00.0: scsi 0:1:0:1: reset logical completed successfully Direct-Access HP LOGICAL VOLUME RAID-0 SSDSmartPathCap+ En+ Exp=1
[10923046.173235] hpsa 0000:03:00.0: scsi 0:0:1:0: removed Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=0
[10923136.937859] hpsa 0000:03:00.0: scsi 0:0:1:0: masked Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=0
```