Page Menu
Home
Phabricator
Search
Configure Global Search
Log In
Paste
P2821
dmesg output
Active
Public
Actions
Authored by
Eevans
on Mar 26 2016, 1:23 PM.
Edit Paste
Archive Paste
View Raw File
Subscribe
Mute Notifications
Award Token
Flag For Later
Tags
None
Referenced Files
F3725985:
Mar 26 2016, 1:24 PM
2016-03-26 13:24:38 (UTC+0)
F3725958:
Mar 26 2016, 1:23 PM
2016-03-26 13:23:17 (UTC+0)
Subscribers
None
[...]
[4401904.509179] hpsa 0000:03:00.0: HP SSD Smart Path: Retrying on standard path.
[4401907.211824] sd 0:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[4401907.211843] sd 0:0:0:0: [sda] Sense Key : Medium Error [current]
[4401907.211849] sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error
[4401907.211852] sd 0:0:0:0: [sda] CDB:
[4401907.211856] Write(10): 2a 00 00 00 08 08 00 00 01 00
[4401907.211869] blk_update_request: critical medium error, dev sda, sector 2056
[4401907.244220] md: super_written gets error=-61, uptodate=0
[4401907.244238] md/raid1:md0: Disk failure on sda1, disabling device.
md/raid1:md0: Operation continuing on 3 devices.
[4401907.308528] RAID1 conf printout:
[4401907.308531] --- wd:3 rd:4
[4401907.308535] disk 0, wo:1, o:0, dev:sda1
[4401907.308537] disk 1, wo:0, o:1, dev:sdb1
[4401907.308539] disk 2, wo:0, o:1, dev:sdc1
[4401907.308541] disk 3, wo:0, o:1, dev:sdd1
[4401907.325791] RAID1 conf printout:
[4401907.325795] --- wd:3 rd:4
[4401907.325798] disk 1, wo:0, o:1, dev:sdb1
[4401907.325799] disk 2, wo:0, o:1, dev:sdc1
[4401907.325801] disk 3, wo:0, o:1, dev:sdd1
[4401932.348955] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated.
[4401966.301902] hpsa 0000:03:00.0: Abort request on C0:B0:T0:L0
[4401966.302019] hpsa 0000:03:00.0: invalid command: LUN:0000004000000000 CDB:0000000000000000f000000000000000
[4401966.302023] hpsa 0000:03:00.0: probably means device no longer present
[4401966.302087] hpsa 0000:03:00.0: invalid command: LUN:0000004000000000 CDB:0000000000000000000000f000000000
[4401966.302089] hpsa 0000:03:00.0: probably means device no longer present
[4401966.302092] hpsa 0000:03:00.0: FAILED abort on device C0:B0:T0:L0
[4401966.302097] hpsa 0000:03:00.0: Abort request on C0:B0:T0:L0
[...]
[4402114.450593] INFO: task md0_raid1:340 blocked for more than 120 seconds.
[4402114.481110] Not tainted 3.19.0-2-amd64 #1
[4402114.502340] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[4402114.538966] md0_raid1 D ffff881034465230 0 340 2 0x00000000
[4402114.538969] ffff881034465230 0000000000000001 ffff88103924f3f0 0000000000014180
[4402114.538971] ffff88103446bfd8 0000000000014180 ffff881034465230 0000000000000246
[4402114.538973] ffff8810349e0800 ffff8810349e0a98 ffff8810349e0a90 ffff8810349e0818
[4402114.538975] Call Trace:
[4402114.539014] [<ffffffffa0126465>] ? md_super_wait+0x45/0x80 [md_mod]
[4402114.539035] [<ffffffff810a8fd0>] ? wait_woken+0x90/0x90
[4402114.539042] [<ffffffffa01268de>] ? md_update_sb+0x23e/0x680 [md_mod]
[4402114.539046] [<ffffffffa0127dea>] ? md_check_recovery+0x18a/0x470 [md_mod]
[4402114.539050] [<ffffffffa004f0e9>] ? raid1d+0x49/0xd50 [raid1]
[4402114.539055] [<ffffffff8154eab7>] ? __schedule+0x7/0x810
[4402114.539057] [<ffffffff8155236a>] ? schedule_timeout+0x21a/0x290
[4402114.539062] [<ffffffffa011fc86>] ? md_thread+0x126/0x130 [md_mod]
[4402114.539064] [<ffffffff810a8fd0>] ? wait_woken+0x90/0x90
[4402114.539067] [<ffffffffa011fb60>] ? md_stop+0x40/0x40 [md_mod]
[4402114.539071] [<ffffffff8108a373>] ? kthread+0xd3/0xf0
[4402114.539073] [<ffffffff8108a2a0>] ? kthread_create_on_node+0x180/0x180
[4402114.539075] [<ffffffff81553418>] ? ret_from_fork+0x58/0x90
[4402114.539077] [<ffffffff8108a2a0>] ? kthread_create_on_node+0x180/0x180
[4402114.539078] INFO: task jbd2/md0-8:369 blocked for more than 120 seconds.
[4402114.570074] Not tainted 3.19.0-2-amd64 #1
[4402114.591358] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[4402114.627887] jbd2/md0-8 D ffff881034464920 0 369 2 0x00000000
[4402114.627889] ffff881034464920 ffffea0081fedf00 ffff881d7847ebe0 0000000000014180
[4402114.627890] ffff8810349d3fd8 0000000000014180 ffff881034464920 0000000000000246
[4402114.627893] ffff88108b1ee100 ffff88108b1ee160 ffff8810349950b8 ffff8810349d3db0
[4402114.627894] Call Trace:
[4402114.627915] [<ffffffffa0176076>] ? jbd2_journal_commit_transaction+0x216/0x1740 [jbd2]
[4402114.627918] [<ffffffff8109c2e6>] ? set_next_entity+0x56/0x70
[4402114.627930] [<ffffffff810a3650>] ? pick_next_task_fair+0x1b0/0x850
[4402114.627935] [<ffffffff810a8fd0>] ? wait_woken+0x90/0x90
[4402114.627938] [<ffffffff810cbd4c>] ? try_to_del_timer_sync+0x4c/0x60
[4402114.627942] [<ffffffffa017acbd>] ? kjournald2+0xcd/0x270 [jbd2]
[4402114.627944] [<ffffffff810a8fd0>] ? wait_woken+0x90/0x90
[4402114.627963] [<ffffffffa017abf0>] ? commit_timeout+0x20/0x20 [jbd2]
[4402114.627964] [<ffffffff8108a373>] ? kthread+0xd3/0xf0
[4402114.627966] [<ffffffff8108a2a0>] ? kthread_create_on_node+0x180/0x180
[4402114.627968] [<ffffffff81553418>] ? ret_from_fork+0x58/0x90
[4402114.627970] [<ffffffff8108a2a0>] ? kthread_create_on_node+0x180/0x180
[4402114.627974] INFO: task jbd2/dm-0-8:611 blocked for more than 120 seconds.
[4402114.659471] Not tainted 3.19.0-2-amd64 #1
[4402114.682356] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[4402114.718771] jbd2/dm-0-8 D ffff88203778c250 0 611 2 0x00000000
[4402114.718927] ffff88203778c250 ffff8810381e6000 ffff88103924f3f0 0000000000014180
[4402114.718928] ffff882036013fd8 0000000000014180 ffff88203778c250 ffffffff8101cc05
[4402114.719081] ffff88207fc74a30 ffff88107ffd6a60 0000000000000002 ffffffff8154fde0
[4402114.719084] Call Trace:
[4402114.719089] [<ffffffff8101cc05>] ? read_tsc+0x5/0x10
[4402114.719090] [<ffffffff8154fde0>] ? bit_wait_timeout+0x80/0x80
[4402114.719093] [<ffffffff8154f5da>] ? io_schedule+0xaa/0x140
[4402114.719094] [<ffffffff8154fe14>] ? bit_wait_io+0x34/0x60
[4402114.719096] [<ffffffff8154f9af>] ? __wait_on_bit+0x5f/0x90
[4402114.719100] [<ffffffff8129d915>] ? queue_unplugged+0x25/0x90
[4402114.719101] [<ffffffff8154fde0>] ? bit_wait_timeout+0x80/0x80
[4402114.719102] [<ffffffff8154fa4e>] ? out_of_line_wait_on_bit+0x6e/0x80
[4402114.719105] [<ffffffff810a9000>] ? autoremove_wake_function+0x30/0x30
[4402114.719110] [<ffffffffa0176d0f>] ? jbd2_journal_commit_transaction+0xeaf/0x1740 [jbd2]
[4402114.719115] [<ffffffffa017acbd>] ? kjournald2+0xcd/0x270 [jbd2]
[4402114.719117] [<ffffffff810a8fd0>] ? wait_woken+0x90/0x90
[4402114.719121] [<ffffffffa017abf0>] ? commit_timeout+0x20/0x20 [jbd2]
[4402114.719124] [<ffffffff8108a373>] ? kthread+0xd3/0xf0
[4402114.719126] [<ffffffff8108a2a0>] ? kthread_create_on_node+0x180/0x180
[4402114.719127] [<ffffffff81553418>] ? ret_from_fork+0x58/0x90
[4402114.719129] [<ffffffff8108a2a0>] ? kthread_create_on_node+0x180/0x180
[...]
4403145.320775] blk_update_request: critical medium error, dev sda, sector 1029788264
[4403145.357401] Aborting journal on device dm-0-8.
[4403145.357491] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1022 pages, ino 143135284; err -30
[4403145.357516] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 34865179; err -30
[4403145.357518] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 34865379; err -30
[4403145.357520] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 34865383; err -30
[4403145.357727] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 143135886; err -30
[4403145.357729] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 143136052; err -30
[4403145.357731] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 143136167; err -30
[4403145.357733] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 34865265; err -30
[4403175.705868] hpsa 0000:03:00.0: Abort request on C0:B0:T0:L0
[4403176.711420] hpsa 0000:03:00.0: device is ready.
[4403176.711429] hpsa 0000:03:00.0: Reset as abort: Device recovered from reset: scsi3addr 0x000000c000000001
[4403176.711433] hpsa 0000:03:00.0: ABORT REQUEST on C0:B0:T0:L0 Tag:0x00000000:00000010 Command:0x2a SN:0x0 REQUEST SUCCEEDED.
[4403176.711456] hpsa 0000:03:00.0: Abort request on C0:B0:T0:L0
[4403176.749374] EXT4-fs error (device dm-0): ext4_journal_check_start:56: Detected aborted journal
[4403176.790437] EXT4-fs (dm-0): Remounting filesystem read-only
[4403192.102023] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated.
[4403199.302030] sd 0:0:0:0: [sda] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[4403199.302050] sd 0:0:0:0: [sda] Sense Key : Medium Error [current]
[4403199.302053] sd 0:0:0:0: [sda] Add. Sense: Unrecovered read error
[4403199.302054] sd 0:0:0:0: [sda] CDB:
[4403199.302056] Write(10): 2a 00 03 9f e4 00 00 00 08 00
[4403199.302061] blk_update_request: critical medium error, dev sda, sector 60810240
[4403199.335848] Buffer I/O error on dev dm-0, logical block 0, lost sync page write
[4403199.369561] EXT4-fs error (device dm-0): ext4_journal_check_start:56: Detected aborted journal
[4403208.354499] hpsa 0000:03:00.0: HP SSD Smart Path: task complete with check condition.
[4403208.354503] hpsa 0000:03:00.0: HP SSD Smart Path: Retrying on standard path.
[4403222.114900] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated.
[4403229.573738] hpsa 0000:03:00.0: Abort request on C0:B0:T0:L0
[4403230.579311] hpsa 0000:03:00.0: device is ready.
[4403230.579321] hpsa 0000:03:00.0: Reset as abort: Device recovered from reset: scsi3addr 0x000000c000000001
[4403230.579325] hpsa 0000:03:00.0: ABORT REQUEST on C0:B0:T0:L0 Tag:0x00000000:00000010 Command:0x2a SN:0x0 REQUEST SUCCEEDED.
[4403252.127419] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated.
[4403253.283431] EXT4-fs error (device dm-0): ext4_journal_check_start:56:
[4403253.283436] EXT4-fs error (device dm-0): ext4_journal_check_start:56: Detected aborted journal
[4403253.354938] Detected aborted journal
[4403253.354957] EXT4-fs (dm-0): ext4_writepages: jbd2_start: 1024 pages, ino 34865378; err -30
[4406814.597501] hpsa 0000:03:00.0: HP SSD Smart Path: task complete with check condition.
[4406814.597509] hpsa 0000:03:00.0: HP SSD Smart Path: Retrying on standard path.
[4406820.818190] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated.
[4410445.560631] hpsa 0000:03:00.0: HP SSD Smart Path: task complete with check condition.
[4410445.560639] hpsa 0000:03:00.0: HP SSD Smart Path: Retrying on standard path.
[4410449.487597] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated.
[4414069.666451] hpsa 0000:03:00.0: HP SSD Smart Path: task complete with check condition.
[4414069.666459] hpsa 0000:03:00.0: HP SSD Smart Path: Retrying on standard path.
[4414078.158926] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated.
[4417690.664284] hpsa 0000:03:00.0: HP SSD Smart Path: task complete with check condition.
[4417690.664292] hpsa 0000:03:00.0: HP SSD Smart Path: Retrying on standard path.
[4417706.829618] hpsa 0000:03:00.0: Direct-Access device c0b0t0l0 updated.
Event Timeline
Eevans
created this paste.
Mar 26 2016, 1:23 PM
2016-03-26 13:23:17 (UTC+0)
Eevans
edited the content of this paste.
(Show Details)
Eevans
changed the title of this paste from untitled to
dmesg output
.
Mar 26 2016, 1:27 PM
2016-03-26 13:27:27 (UTC+0)
Eevans
mentioned this in
T130254: Investigate recent OOM events on restbase2004
.
Mar 26 2016, 1:37 PM
2016-03-26 13:37:36 (UTC+0)
Eevans
mentioned this in
T130990: restbase2004.codfw.wmnet: Failed disk/RAID
.
Mar 26 2016, 1:42 PM
2016-03-26 13:42:59 (UTC+0)
Log In to Comment