logstash1006 seems to have a failed disk. Icinga alerts that /var/lib/elasticsearch is not accessible. dmesg indicates IO errors:
[4035950.366928] sd 1:0:0:0: [sdb] tag#10 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK [4035950.366934] sd 1:0:0:0: [sdb] tag#10 CDB: Read(10) 28 00 00 00 08 08 00 00 08 00 [4035950.366950] blk_update_request: I/O error, dev sdb, sector 2056
@Cmjohnson could you have a look and see if we have a spare for a 1Tb spinning disk?