new hardware has been received as per {T126049}, next steps are to rack / install / burn in the hardware
Description
Description
Details
Details
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • mobrovac | T112648 enable restbase syslog/file logging | |||
| Resolved | fgiunchedi | T113714 Separate /var on restbase | |||
| Resolved | fgiunchedi | T121575 Expand SSD space in Cassandra cluster | |||
| Resolved | RobH | T125842 normalize eqiad restbase cluster - replace restbase1001-1006 | |||
| Resolved | Eevans | T134016 RESTBase Cassandra cluster: Increase instance count to 3 | |||
| Invalid | fgiunchedi | T85451 scale graphite deployment (tracking) | |||
| Resolved | RobH | T126253 additional graphite machines request, 1x per DC | |||
| Resolved | RobH | T131184 eqiad: (2) Relevance forge servers | |||
| Unknown Object (Task) | |||||
| Resolved | Cmjohnson | T130752 reclaim restbase1001-1006 to spares | |||
| Resolved | Cmjohnson | T128107 install restbase1010-restbase1015 |
Event Timeline
Comment Actions
we'll be starting with row 'A', same row as restbase1001 / restbase1002 / restbase1007, at two machines per row
Comment Actions
chatting with @Cmjohnson on IRC, we'll need to install the new machines with 5x ssd from the beginning, there should be 12x + 3x SSD in eqiad. IOW we have SSD for 3x machines to start with, the remaining 18x SSD will come from restbase1001-1006 as they are progressively decommissioned.
Comment Actions
I guess if we're wanting to get ahead of the curve, the third machine could be stood up in rack 'B'.
Comment Actions
Without taking from existing servers (restbae1001-6). I am able to stand up
2 machines (restbase1010-1011). To get the next 2(1012-1013) servers going
we'll need the ssds from restbase1001-1002.
Comment Actions
Current status is 1010 and 1011 are ready for installs, everything but
identifying the partman recipe is complete
completed: racked, cabled, switch cfg, dhcp, dns, confirmed virtual console
Comment Actions
Change 273912 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: add restbase1010-restbase1015 to site/install_server
Comment Actions
Change 273912 merged by Filippo Giunchedi:
cassandra: add restbase1010-restbase1015 to site/install_server
Comment Actions
thanks @Cmjohnson ! I could successfully install restbase1010 after fixing partman in https://gerrit.wikimedia.org/r/273912 though restbase1011 doesn't seem to be able to pxe-boot yet (stuck in a loop boot)
Comment Actions
Change 274105 had a related patch set uploaded (by Filippo Giunchedi):
add restbase101[0-5] cassandra instances
Comment Actions
Change 274107 had a related patch set uploaded (by Filippo Giunchedi):
hieradata: add restbase101[0-5] instances to cassandra seeds
Comment Actions
Change 274107 merged by Filippo Giunchedi:
hieradata: add restbase101[0-5] instances to cassandra seeds
Comment Actions
Change 274133 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: add restbase101[0-5] instances
Comment Actions
issue was bios set to boot as UEFI, when set to bios legacy I was able to pxe-boot it and run debian-installer
Comment Actions
Change 274724 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: add restbase1010-b instance
Comment Actions
Mentioned in SAL [2016-03-03T17:58:04Z] <urandom> increasing stream throughput for restbase1010-b.eqiad.wmnet boostrap by 25mbps (5x5) : T128107 T95253
Comment Actions
Mentioned in SAL [2016-03-03T18:14:27Z] <urandom> lowering outbound stream throughput limit on restbase1010-a.eqiad.wmnet to 25mbps : T128107 T95253
Comment Actions
status update:
restbase1010 is in service with two instances
restbase1011 is bootstrapping its second instance (ETA 7-8h)
after that's done we can start decommissioning restbase1001 / restbase1002
Comment Actions
Change 277206 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: deprovision restbase100[12]
Comment Actions
Change 277208 had a related patch set uploaded (by Filippo Giunchedi):
deprovision restbase100[12]
Comment Actions
Change 277210 had a related patch set uploaded (by Filippo Giunchedi):
conftool: swap restbase100[12] with restbase101[01]
Comment Actions
Chris, restbase100[12] have been halted and they can be repurposed modulo SSDs which we'll need for other restbase machines in this ticket, let me know when ssds have been installed in restbase101[34]
Comment Actions
Change 277210 merged by Filippo Giunchedi:
conftool: swap restbase100[12] with restbase101[01]
Comment Actions
@fgiunchedi restbase101[23] have ssds now, enabled on the switch and are ready for you to install.
Comment Actions
thanks @Cmjohnson ! I've halted both restbase1003 and restbase1004, should be good to be reclaimed.
once that's done we can move onto the last row with restbase1014 / restbase1015 !
Comment Actions
@fgiunchedi restbase1014 is ready for you. I cannot do restbase1015 until both restabse1005/1006 are taken offline. I do not have enough disks to do both.
Comment Actions
thanks @Cmjohnson ! I don't see the port configured in asw-d-eqiad nor port assignment in phab or racktables though, can you help?
Comment Actions
@Cmjohnson I thought we had enough disks to deploy two new nodes at a time for all three racks (providing of course that we decommissioned two nodes and cannibalized the disks each time)? Was that not the case, or did something change?
Comment Actions
No, we do not have enough disks. we are adding 5 each and removing 3. even if i used the remaining 3 spare i am still 1 disk short of being able to populate the last restbase
Comment Actions
For posterity, here's @Eevans' maths as to why we are short of disks:
(15:32:20) urandom: actually, i think i get itt (15:32:34) mobrovac: have we miscalculated the number of disks? (15:32:43) urandom: we put two machines up, requiring 10 disks (5 each) (15:33:04) urandom: then we decomm'd 2, recoving 6 (3 each) (15:33:17) urandom: then we did the same a second time (15:33:24) urandom: using 10, and recovering 6 (15:33:41) mobrovac: but we ordered the extra disks a while ago, haven't we? (15:34:09) urandom: let's see, we ordered enough to take use to 5 disks, and we had already done...3 of them? (15:34:50) urandom: so we had 12 disks for expansion (6*2)? (15:34:59) urandom: then some spares, I guess (15:35:36) urandom: godog: does that sound right? (15:37:04) mobrovac: we already had 9 nodes at the time of the order, i think (15:37:22) urandom: yes, and i think we had already expanded to 5 disks in 3 of them (15:37:40) urandom: 7, 8, and 9, if my recollection is correct (15:37:45) mobrovac: so i think we ordered 9*2 disks, not 6*2 (15:37:50) mobrovac: yes, those were the first ones (15:38:15) urandom: yes, but at the time we started standing up new nodes, we had already used 2*3 in nodes 7,8, and 9 (15:38:22) urandom: leaving us with 6*2 (15:38:51) urandom: 2 disks each, for the remaining 6 nodes (15:39:49) mobrovac: ok (15:40:03) mobrovac: and now, we are moving 01:06 to 10:15 (15:40:14) mobrovac: my maths says that's the same number of machines (15:40:24) urandom: the new machines had no disks (15:40:31) mobrovac: ah! (15:40:34) urandom: we had 12 to start off with (15:40:51) urandom: yeah (15:40:52) mobrovac: assumption fail (15:40:53) urandom: :) (15:40:53) mobrovac: :) (15:41:21) urandom: so we robbed Peter to pay Paul until our Peter hurt (15:41:44) urandom: (thank you, i'm here all week) (15:41:59) urandom: (don't forget to tip your waitstaff)
Comment Actions
tl;dr: We don't have enough spares to temporarily stand up an extra node with 5T storage.
Comment Actions
Change 282361 had a related patch set uploaded (by Filippo Giunchedi):
cassandra: add restbase1014-a
Comment Actions
@Cmjohnson I've deprovisioned restbase1005 and restbase1006 and both are shutdown, should be enough disks to get restbase1015 going now, thanks!
Comment Actions
the bootstrap didn't complete as expected with cassandra reporting corrupted sstables in logs
root@restbase1015:/var/log/cassandra# grep ERROR system-a.log ERROR [main] 2016-04-22 13:36:46,817 CassandraDaemon.java:393 - Fatal configuration error ERROR [MigrationStage:1] 2016-04-22 13:39:51,309 FailureDetector.java:223 - unknown endpoint /10.64.32.205 ERROR [MigrationStage:1] 2016-04-22 13:39:51,367 FailureDetector.java:223 - unknown endpoint /10.64.48.135 ERROR [CompactionExecutor:444] 2016-04-22 18:23:24,443 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:444,1,main] ERROR [CompactionExecutor:444] 2016-04-22 18:23:24,445 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:19] 2016-04-22 19:25:18,375 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:19,1,main] ERROR [CompactionExecutor:19] 2016-04-22 19:25:18,377 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:3] 2016-04-22 19:37:41,415 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:3,1,main] ERROR [CompactionExecutor:3] 2016-04-22 19:37:41,418 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:10] 2016-04-22 20:08:20,432 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:10,1,main] ERROR [CompactionExecutor:10] 2016-04-22 20:08:20,436 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:1] 2016-04-22 20:39:50,433 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:1,1,main] ERROR [CompactionExecutor:1] 2016-04-22 20:39:50,435 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:6] 2016-04-22 21:08:07,790 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:6,1,main] ERROR [CompactionExecutor:6] 2016-04-22 21:08:07,793 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:8] 2016-04-22 21:38:01,460 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:8,1,main] ERROR [CompactionExecutor:8] 2016-04-22 21:38:01,462 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:7] 2016-04-22 22:08:02,257 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:7,1,main] ERROR [CompactionExecutor:7] 2016-04-22 22:08:02,259 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:9] 2016-04-22 22:38:24,659 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:9,1,main] ERROR [CompactionExecutor:9] 2016-04-22 22:38:24,684 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:10] 2016-04-22 23:08:02,484 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:10,1,main] ERROR [CompactionExecutor:10] 2016-04-22 23:08:02,493 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:9] 2016-04-22 23:37:46,427 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:9,1,main] ERROR [CompactionExecutor:9] 2016-04-22 23:37:46,430 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:3] 2016-04-23 00:08:20,325 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:3,1,main] ERROR [CompactionExecutor:3] 2016-04-23 00:08:20,327 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:10] 2016-04-23 00:38:10,952 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:10,1,main] ERROR [CompactionExecutor:10] 2016-04-23 00:38:10,954 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:26] 2016-04-23 01:19:53,937 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:26,1,main] ERROR [CompactionExecutor:26] 2016-04-23 01:19:53,939 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:1] 2016-04-23 01:39:01,211 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:1,1,main] ERROR [CompactionExecutor:1] 2016-04-23 01:39:01,214 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:10] 2016-04-23 02:07:55,352 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:10,1,main] ERROR [CompactionExecutor:10] 2016-04-23 02:07:55,354 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:8] 2016-04-23 02:37:51,488 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:8,1,main] ERROR [CompactionExecutor:8] 2016-04-23 02:37:51,490 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:4] 2016-04-23 03:08:17,969 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:4,1,main] ERROR [CompactionExecutor:4] 2016-04-23 03:08:17,972 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:4] 2016-04-23 03:37:36,780 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:4,1,main] ERROR [CompactionExecutor:4] 2016-04-23 03:37:36,782 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:9] 2016-04-23 04:08:18,449 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:9,1,main] ERROR [CompactionExecutor:9] 2016-04-23 04:08:18,452 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:10] 2016-04-23 04:37:32,204 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:10,1,main] ERROR [CompactionExecutor:10] 2016-04-23 04:37:32,207 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:9] 2016-04-23 05:07:34,578 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:9,1,main] ERROR [CompactionExecutor:9] 2016-04-23 05:07:34,582 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:6] 2016-04-23 05:37:43,637 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:6,1,main] ERROR [CompactionExecutor:6] 2016-04-23 05:37:43,640 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:7] 2016-04-23 06:08:04,476 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:7,1,main] ERROR [CompactionExecutor:7] 2016-04-23 06:08:04,479 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:7] 2016-04-23 06:37:42,678 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:7,1,main] ERROR [CompactionExecutor:7] 2016-04-23 06:37:42,681 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:7] 2016-04-23 07:07:34,124 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:7,1,main] ERROR [CompactionExecutor:7] 2016-04-23 07:07:34,127 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:9] 2016-04-23 07:37:26,231 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:9,1,main] ERROR [CompactionExecutor:9] 2016-04-23 07:37:26,233 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:10] 2016-04-23 08:07:38,763 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:10,1,main] ERROR [CompactionExecutor:10] 2016-04-23 08:07:38,765 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" ERROR [CompactionExecutor:1] 2016-04-23 08:37:21,887 CassandraDaemon.java:229 - Exception in thread Thread[CompactionExecutor:1,1,main] ERROR [CompactionExecutor:1] 2016-04-23 08:37:21,891 FileUtils.java:447 - Exiting forcefully due to file system exception on startup, disk failure policy "stop" root@restbase1015:/var/log/cassandra#
Apr 22 13:37:02 restbase1015 kernel: [ 8777.768915] ip6_tables: (C) 2000-2006 Netfilter Core Team Apr 22 13:45:35 restbase1015 kernel: [ 9291.566670] perf interrupt took too long (2514 > 2500), lowering kernel.perf_event_max_sample_rate to 50000 Apr 22 13:52:22 restbase1015 kernel: [ 9698.615155] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff8810340f6040Tag:0x00000000:000000f0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 13:52:22 restbase1015 kernel: [ 9698.615161] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 13:52:43 restbase1015 kernel: [ 9719.557149] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 13:52:43 restbase1015 kernel: [ 9719.557198] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff8810340f6040Tag:0x00000000:000000f0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, FAILED Apr 22 13:52:43 restbase1015 kernel: [ 9719.557203] hpsa 0000:03:00.0: scsi 0:0:2:0: FAILED to abort command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 13:52:43 restbase1015 kernel: [ 9719.557209] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff8820369c64c0Tag:0x00000000:00000130 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 13:52:43 restbase1015 kernel: [ 9719.557211] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 13:52:43 restbase1015 kernel: [ 9719.557306] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff8820369c64c0Tag:0x00000000:00000130 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 13:52:43 restbase1015 kernel: [ 9719.587314] hpsa 0000:03:00.0: CDB 2a001c93e67000000800000000000000 was aborted with status 0x0 Apr 22 13:52:43 restbase1015 kernel: [ 9719.587448] hpsa 0000:03:00.0: scsi 0:0:2:0: resetting physical Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 13:52:43 restbase1015 kernel: [ 9719.587615] hpsa 0000:03:00.0: scsi 0:0:2:0: reset physical completed successfully Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 13:55:43 restbase1015 kernel: [ 9899.697940] hpsa 0000:03:00.0: scsi 0:0:3:0 Aborting command ffff882038343940Tag:0x00000000:000001a0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 13:55:43 restbase1015 kernel: [ 9899.697946] hpsa 0000:03:00.0: scsi 0:0:3:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 13:55:43 restbase1015 kernel: [ 9899.698069] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 13:55:43 restbase1015 kernel: [ 9899.698085] hpsa 0000:03:00.0: scsi 0:0:3:0 Aborting command ffff882038343940Tag:0x00000000:000001a0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 13:55:43 restbase1015 kernel: [ 9899.698099] hpsa 0000:03:00.0: scsi 0:0:3:0 Aborting command ffff88203579d500Tag:0x00000000:00000190 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 13:55:43 restbase1015 kernel: [ 9899.698107] hpsa 0000:03:00.0: scsi 0:0:3:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 13:55:43 restbase1015 kernel: [ 9899.698176] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 13:55:43 restbase1015 kernel: [ 9899.698187] hpsa 0000:03:00.0: scsi 0:0:3:0 Aborting command ffff88203579d500Tag:0x00000000:00000190 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 13:55:43 restbase1015 kernel: [ 9899.698197] hpsa 0000:03:00.0: scsi 0:0:3:0 Aborting command ffff880c8d4b2b00Tag:0x00000000:00000160 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 13:55:43 restbase1015 kernel: [ 9899.698206] hpsa 0000:03:00.0: scsi 0:0:3:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 13:55:43 restbase1015 kernel: [ 9899.698265] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 13:55:43 restbase1015 kernel: [ 9899.698273] hpsa 0000:03:00.0: scsi 0:0:3:0 Aborting command ffff880c8d4b2b00Tag:0x00000000:00000160 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 13:55:43 restbase1015 kernel: [ 9899.698281] hpsa 0000:03:00.0: scsi 0:0:3:0 Aborting command ffff8810397127c0Tag:0x00000000:00000130 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 13:55:43 restbase1015 kernel: [ 9899.698285] hpsa 0000:03:00.0: scsi 0:0:3:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 13:55:43 restbase1015 kernel: [ 9899.698367] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 13:55:43 restbase1015 kernel: [ 9899.698395] hpsa 0000:03:00.0: scsi 0:0:3:0 Aborting command ffff8810397127c0Tag:0x00000000:00000130 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 13:55:43 restbase1015 kernel: [ 9899.698415] hpsa 0000:03:00.0: scsi 0:0:3:0 Aborting command ffff88103801a340Tag:0x00000000:000000e0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 13:55:43 restbase1015 kernel: [ 9899.698426] hpsa 0000:03:00.0: scsi 0:0:3:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 13:55:43 restbase1015 kernel: [ 9899.698503] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 13:55:43 restbase1015 kernel: [ 9899.698517] hpsa 0000:03:00.0: scsi 0:0:3:0 Aborting command ffff88103801a340Tag:0x00000000:000000e0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 13:55:43 restbase1015 kernel: [ 9899.698527] hpsa 0000:03:00.0: scsi 0:0:3:0 Aborting command ffff8810375c8b00Tag:0x00000000:000000d0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 13:55:43 restbase1015 kernel: [ 9899.698532] hpsa 0000:03:00.0: scsi 0:0:3:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 13:56:07 restbase1015 kernel: [ 9923.558583] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 13:56:07 restbase1015 kernel: [ 9923.558650] hpsa 0000:03:00.0: scsi 0:0:3:0 Aborting command ffff8810375c8b00Tag:0x00000000:000000d0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, FAILED Apr 22 13:56:07 restbase1015 kernel: [ 9923.558657] hpsa 0000:03:00.0: scsi 0:0:3:0: FAILED to abort command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 13:56:07 restbase1015 kernel: [ 9923.558664] hpsa 0000:03:00.0: scsi 0:0:3:0 Aborting command ffff8810392a0840Tag:0x00000000:00000140 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 13:56:07 restbase1015 kernel: [ 9923.558666] hpsa 0000:03:00.0: scsi 0:0:3:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 13:56:07 restbase1015 kernel: [ 9923.558710] hpsa 0000:03:00.0: scsi 0:0:3:0 Aborting command ffff8810392a0840Tag:0x00000000:00000140 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 13:56:07 restbase1015 kernel: [ 9923.570153] hpsa 0000:03:00.0: CDB 2a001ca181c000000800000000000000 was aborted with status 0x0 Apr 22 13:56:07 restbase1015 kernel: [ 9923.584672] hpsa 0000:03:00.0: scsi 0:0:3:0: resetting physical Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 13:56:07 restbase1015 kernel: [ 9923.584828] hpsa 0000:03:00.0: scsi 0:0:3:0: reset physical completed successfully Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 13:59:25 restbase1015 kernel: [10121.990085] perf interrupt took too long (5552 > 5000), lowering kernel.perf_event_max_sample_rate to 25000 Apr 22 17:16:12 restbase1015 kernel: [21928.402261] hpsa 0000:03:00.0: scsi 0:0:1:0 Aborting command ffff882035938340Tag:0x00000000:000000f0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 17:16:12 restbase1015 kernel: [21928.402270] hpsa 0000:03:00.0: scsi 0:0:1:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 17:16:12 restbase1015 kernel: [21928.402387] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 17:16:12 restbase1015 kernel: [21928.402404] hpsa 0000:03:00.0: scsi 0:0:1:0 Aborting command ffff882035938340Tag:0x00000000:000000f0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 17:16:12 restbase1015 kernel: [21928.402425] hpsa 0000:03:00.0: scsi 0:0:1:0 Aborting command ffff8820369d2c40Tag:0x00000000:000000e0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 17:16:12 restbase1015 kernel: [21928.402437] hpsa 0000:03:00.0: scsi 0:0:1:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 17:16:12 restbase1015 kernel: [21928.402499] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 17:16:12 restbase1015 kernel: [21928.402513] hpsa 0000:03:00.0: scsi 0:0:1:0 Aborting command ffff8820369d2c40Tag:0x00000000:000000e0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 17:16:12 restbase1015 kernel: [21928.402526] hpsa 0000:03:00.0: scsi 0:0:1:0 Aborting command ffff881039181380Tag:0x00000000:000000d0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 17:16:12 restbase1015 kernel: [21928.402530] hpsa 0000:03:00.0: scsi 0:0:1:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 17:16:33 restbase1015 kernel: [21949.642718] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 17:16:33 restbase1015 kernel: [21949.642774] hpsa 0000:03:00.0: scsi 0:0:1:0 Aborting command ffff881039181380Tag:0x00000000:000000d0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, FAILED Apr 22 17:16:33 restbase1015 kernel: [21949.642785] hpsa 0000:03:00.0: scsi 0:0:1:0: FAILED to abort command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 17:16:33 restbase1015 kernel: [21949.642797] hpsa 0000:03:00.0: scsi 0:0:1:0 Aborting command ffff880c58990b00Tag:0x00000000:000001b0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 17:16:33 restbase1015 kernel: [21949.642802] hpsa 0000:03:00.0: scsi 0:0:1:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 17:16:33 restbase1015 kernel: [21949.643059] hpsa 0000:03:00.0: scsi 0:0:1:0 Aborting command ffff880c58990b00Tag:0x00000000:000001b0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 17:16:33 restbase1015 kernel: [21949.659824] hpsa 0000:03:00.0: CDB 2a000016e8b800001000000000000000 was aborted with status 0x0 Apr 22 17:16:33 restbase1015 kernel: [21949.676870] hpsa 0000:03:00.0: scsi 0:0:1:0: resetting physical Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 17:16:33 restbase1015 kernel: [21949.677027] hpsa 0000:03:00.0: scsi 0:0:1:0: reset physical completed successfully Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 19:04:43 restbase1015 kernel: [28439.255762] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff88003644c540Tag:0x00000000:00000260 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 19:04:43 restbase1015 kernel: [28439.255771] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 19:04:43 restbase1015 kernel: [28439.255855] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 19:04:43 restbase1015 kernel: [28439.255864] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff88003644c540Tag:0x00000000:00000260 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 19:04:43 restbase1015 kernel: [28439.255883] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff881039181380Tag:0x00000000:000000e0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 19:04:43 restbase1015 kernel: [28439.255887] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 19:04:43 restbase1015 kernel: [28439.255941] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 19:04:43 restbase1015 kernel: [28439.255946] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff881039181380Tag:0x00000000:000000e0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 19:04:43 restbase1015 kernel: [28439.255956] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff880c694e20c0Tag:0x00000000:00000250 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 19:04:43 restbase1015 kernel: [28439.255960] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 19:04:43 restbase1015 kernel: [28439.256013] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 19:04:43 restbase1015 kernel: [28439.256029] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff880c694e20c0Tag:0x00000000:00000250 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 19:04:43 restbase1015 kernel: [28439.256033] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff880c5b2330c0Tag:0x00000000:00000240 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 19:04:43 restbase1015 kernel: [28439.256035] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 19:04:43 restbase1015 kernel: [28439.256085] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 19:04:43 restbase1015 kernel: [28439.256087] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff880c5b2330c0Tag:0x00000000:00000240 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 19:04:43 restbase1015 kernel: [28439.256090] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff881037ee21c0Tag:0x00000000:00000230 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 19:04:43 restbase1015 kernel: [28439.256092] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 19:04:43 restbase1015 kernel: [28439.256142] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 19:04:43 restbase1015 kernel: [28439.256144] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff881037ee21c0Tag:0x00000000:00000230 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 19:04:43 restbase1015 kernel: [28439.256147] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff881037f151c0Tag:0x00000000:00000220 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 19:04:43 restbase1015 kernel: [28439.256149] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 19:04:43 restbase1015 kernel: [28439.256199] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 19:04:43 restbase1015 kernel: [28439.256201] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff881037f151c0Tag:0x00000000:00000220 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 19:04:43 restbase1015 kernel: [28439.256205] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff88045d75c0c0Tag:0x00000000:00000210 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 19:04:43 restbase1015 kernel: [28439.256206] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 19:04:43 restbase1015 kernel: [28439.256256] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 19:04:43 restbase1015 kernel: [28439.256258] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff88045d75c0c0Tag:0x00000000:00000210 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 19:04:43 restbase1015 kernel: [28439.256261] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff88100925e380Tag:0x00000000:000000d0 CDBLen: 10 CDB: 0x2a08... SN: 0x0 BEING SENT Apr 22 19:04:43 restbase1015 kernel: [28439.256262] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 19:04:43 restbase1015 kernel: [28439.256312] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 19:04:43 restbase1015 kernel: [28439.256314] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff88100925e380Tag:0x00000000:000000d0 CDBLen: 10 CDB: 0x2a08... SN: 0x0 SENT, SUCCESS Apr 22 19:04:43 restbase1015 kernel: [28439.256318] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff880c68bca4c0Tag:0x00000000:00000100 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 19:04:43 restbase1015 kernel: [28439.256319] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 19:04:43 restbase1015 kernel: [28439.256391] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 19:04:43 restbase1015 kernel: [28439.256421] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff880c68bca4c0Tag:0x00000000:00000100 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 19:04:44 restbase1015 kernel: [28440.163608] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff8803cd0e2340Tag:0x00000000:00000110 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 22 19:04:44 restbase1015 kernel: [28440.163616] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 22 19:04:44 restbase1015 kernel: [28440.163701] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 22 19:04:44 restbase1015 kernel: [28440.163727] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff8803cd0e2340Tag:0x00000000:00000110 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 22 22:46:00 restbase1015 kernel: [41715.948785] perf interrupt took too long (10440 > 10000), lowering kernel.perf_event_max_sample_rate to 12500 Apr 23 01:38:15 restbase1015 kernel: [52050.854038] blk_update_request: I/O error, dev sdb, sector 0 Apr 23 01:55:08 restbase1015 kernel: [53063.795385] hpsa 0000:03:00.0: scsi 0:0:4:0 Aborting command ffff88203579d500Tag:0x00000000:000000d0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 23 01:55:08 restbase1015 kernel: [53063.795395] hpsa 0000:03:00.0: scsi 0:0:4:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 23 01:55:33 restbase1015 kernel: [53088.847859] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 23 01:55:33 restbase1015 kernel: [53088.847886] hpsa 0000:03:00.0: scsi 0:0:4:0 Aborting command ffff88203579d500Tag:0x00000000:000000d0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, FAILED Apr 23 01:55:33 restbase1015 kernel: [53088.847899] hpsa 0000:03:00.0: scsi 0:0:4:0: FAILED to abort command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 23 01:55:33 restbase1015 kernel: [53088.847912] hpsa 0000:03:00.0: scsi 0:0:4:0 Aborting command ffff88203579dc80Tag:0x00000000:000001e0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 23 01:55:33 restbase1015 kernel: [53088.847918] hpsa 0000:03:00.0: scsi 0:0:4:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 23 01:55:33 restbase1015 kernel: [53088.847965] hpsa 0000:03:00.0: scsi 0:0:4:0 Aborting command ffff88203579dc80Tag:0x00000000:000001e0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 23 01:55:33 restbase1015 kernel: [53088.858747] hpsa 0000:03:00.0: CDB 2a00008f5ab800000800000000000000 was aborted with status 0x0 Apr 23 01:55:33 restbase1015 kernel: [53088.864985] hpsa 0000:03:00.0: scsi 0:0:4:0: resetting physical Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 23 01:55:33 restbase1015 kernel: [53088.865143] hpsa 0000:03:00.0: scsi 0:0:4:0: reset physical completed successfully Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 23 06:25:01 restbase1015 kernel: [69256.427117] Process accounting resumed Apr 23 07:54:40 restbase1015 kernel: [74635.335275] hpsa 0000:03:00.0: scsi 0:0:4:0 Aborting command ffff8810385c2340Tag:0x00000000:000001e0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 23 07:54:40 restbase1015 kernel: [74635.335280] hpsa 0000:03:00.0: scsi 0:0:4:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 23 07:55:17 restbase1015 kernel: [74672.983702] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 23 07:55:17 restbase1015 kernel: [74672.983741] hpsa 0000:03:00.0: scsi 0:0:4:0 Aborting command ffff8810385c2340Tag:0x00000000:000001e0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, FAILED Apr 23 07:55:17 restbase1015 kernel: [74672.983749] hpsa 0000:03:00.0: scsi 0:0:4:0: FAILED to abort command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 23 07:55:17 restbase1015 kernel: [74672.983760] hpsa 0000:03:00.0: scsi 0:0:4:0 Aborting command ffff880c687c53c0Tag:0x00000000:000000d0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 23 07:55:17 restbase1015 kernel: [74672.983765] hpsa 0000:03:00.0: scsi 0:0:4:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 23 07:55:17 restbase1015 kernel: [74672.983816] hpsa 0000:03:00.0: scsi 0:0:4:0 Aborting command ffff880c687c53c0Tag:0x00000000:000000d0 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 23 07:55:17 restbase1015 kernel: [74672.999747] hpsa 0000:03:00.0: CDB 2a00008f5ab000001000000000000000 was aborted with status 0x0 Apr 23 07:55:17 restbase1015 kernel: [74672.999875] hpsa 0000:03:00.0: scsi 0:0:4:0: resetting physical Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 23 07:55:17 restbase1015 kernel: [74673.000047] hpsa 0000:03:00.0: scsi 0:0:4:0: reset physical completed successfully Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 23 18:32:56 restbase1015 kernel: [112930.483803] perf interrupt took too long (20259 > 20000), lowering kernel.perf_event_max_sample_rate to 6250 Apr 24 00:02:01 restbase1015 kernel: [132675.218667] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff8820369d2c40Tag:0x00000000:000000f0 CDBLen: 10 CDB: 0x2a08... SN: 0x0 BEING SENT Apr 24 00:02:01 restbase1015 kernel: [132675.218677] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 24 00:02:01 restbase1015 kernel: [132675.218781] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 24 00:02:01 restbase1015 kernel: [132675.218811] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff8820369d2c40Tag:0x00000000:000000f0 CDBLen: 10 CDB: 0x2a08... SN: 0x0 SENT, SUCCESS Apr 24 00:02:01 restbase1015 kernel: [132675.218831] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff881a0bd4d340Tag:0x00000000:00000170 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 24 00:02:01 restbase1015 kernel: [132675.218844] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 24 00:02:01 restbase1015 kernel: [132675.218912] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 24 00:02:01 restbase1015 kernel: [132675.218925] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff881a0bd4d340Tag:0x00000000:00000170 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 24 00:02:01 restbase1015 kernel: [132675.218935] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff882035730500Tag:0x00000000:00000160 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 24 00:02:01 restbase1015 kernel: [132675.218939] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 24 00:02:01 restbase1015 kernel: [132675.218993] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 24 00:02:01 restbase1015 kernel: [132675.219002] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff882035730500Tag:0x00000000:00000160 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 24 00:02:06 restbase1015 kernel: [132680.210570] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff882038343040Tag:0x00000000:00000130 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 24 00:02:06 restbase1015 kernel: [132680.210577] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 24 00:02:06 restbase1015 kernel: [132680.210662] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 24 00:02:06 restbase1015 kernel: [132680.210690] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff882038343040Tag:0x00000000:00000130 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS Apr 24 00:02:06 restbase1015 kernel: [132680.210708] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff882038343940Tag:0x00000000:00000120 CDBLen: 10 CDB: 0x2a00... SN: 0x0 BEING SENT Apr 24 00:02:06 restbase1015 kernel: [132680.210721] hpsa 0000:03:00.0: scsi 0:0:2:0: Aborting command Direct-Access ATA Samsung SSD 850 PHYS DRV SSDSmartPathCap- En- Exp=1 Apr 24 00:02:06 restbase1015 kernel: [132680.210785] hpsa 0000:03:00.0: CDB 00000000000000000000000000000000 was aborted with status 0x0 Apr 24 00:02:06 restbase1015 kernel: [132680.210792] hpsa 0000:03:00.0: scsi 0:0:2:0 Aborting command ffff882038343940Tag:0x00000000:00000120 CDBLen: 10 CDB: 0x2a00... SN: 0x0 SENT, SUCCESS
the raid controller though is in HBA mode, not raid, which might be the reason for failures. I've saved the logs and will reconfigure the raid and reimage
Comment Actions
Change 285420 had a related patch set uploaded (by Eevans):
cassandra: add restbase1015-b