Paste P2092

labnet1002 kernel stack trace
ActivePublic
Actions

Authored by akosiaris on Sep 25 2015, 11:24 AM.

Tags

Referenced Files

	F2634162: labnet1002 kernel stack trace
	Sep 25 2015, 11:27 AM

	F2634158: labnet1002 kernel stack trace
	Sep 25 2015, 11:24 AM

Subscribers

None

	[5147827.218462] BUG: soft lockup - CPU#2 stuck for 22s! [nova-api:3184]
	[5147827.224887] Modules linked in: nfnetlink bluetooth ebt_redirect ebt_ip ebtable_nat ebtables xt_CHECKSUM xt_conntrack bridge xt_nat xt_tcpudp iptable_filter iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables ipmi_devintf intel_powerclamp coretemp 8021q garp stp mrp kvm llc gpio_ich dcdbas crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul joydev glue_helper ablk_helper cryptd i7core_edac serio_raw ipmi_si shpchp edac_core acpi_power_meter lpc_ich mac_hid lp parport hid_generic usbhid hid mptsas mptscsih bnx2x mptbase psmouse mdio pata_acpi scsi_transport_sas libcrc32c
	[5147827.284008] CPU: 2 PID: 3184 Comm: nova-api Tainted: G W 3.13.0-59-generic #98-Ubuntu
	[5147827.292944] Hardware name: Dell Inc. PowerEdge R610/0F0XJ6, BIOS 6.1.0 10/18/2011
	[5147827.300582] task: ffff8802245b1800 ti: ffff880418a76000 task.ti: ffff880418a76000
	[5147827.308221] RIP: 0010:[<ffffffff810dc202>] [<ffffffff810dc202>] generic_exec_single+0x82/0xb0
	[5147827.316994] RSP: 0000:ffff880418a77a20 EFLAGS: 00000202
	[5147827.322464] RAX: 0000000000000020 RBX: 0000000000000001 RCX: 0000000000000010
	[5147827.329754] RDX: 0000000000000100 RSI: 0000000000000000 RDI: 0000000000000286
	[5147827.337044] RBP: ffff880418a77a50 R08: ffffffff8180af28 R09: ffff880421c6d060
	[5147827.344336] R10: ffff88042fff9f00 R11: ffffea000d95ee60 R12: ffffffff810a6cf3
	[5147827.351626] R13: ffff880418a77af0 R14: 0000000000000017 R15: ffff880227d73180
	[5147827.358916] FS: 00007ff62c9e2740(0000) GS:ffff88042fc20000(0000) knlGS:0000000000000000
	[5147827.367159] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
	[5147827.373065] CR2: 0000000001943b18 CR3: 000000041ae62000 CR4: 00000000000007e0
	[5147827.380354] Stack:
	[5147827.382531] ffff880227cf3c40 000000000000000f 0000000000000002 ffffffff81d144c0
	[5147827.390116] ffffffff8105c800 0000000000000301 ffff880418a77ac8 ffffffff810dc315
	[5147827.397701] ffff88042fffc040 ffff88042fff9f68 ffff880422189a70 ffff880227cf3c40
	[5147827.405289] Call Trace:
	[5147827.407902] [<ffffffff8105c800>] ? leave_mm+0x80/0x80
	[5147827.413200] [<ffffffff810dc315>] smp_call_function_single+0xe5/0x190
	[5147827.419799] [<ffffffff8105c800>] ? leave_mm+0x80/0x80
	[5147827.425099] [<ffffffff810dc746>] smp_call_function_many+0x286/0x2d0
	[5147827.431611] [<ffffffff8105c800>] ? leave_mm+0x80/0x80
	[5147827.436911] [<ffffffff8105c957>] native_flush_tlb_others+0x37/0x40
	[5147827.443335] [<ffffffff8105cc56>] flush_tlb_page+0x56/0xa0
	[5147827.448981] [<ffffffff8118a7d8>] ptep_clear_flush+0x48/0x60
	[5147827.454800] [<ffffffff81184ddf>] try_to_unmap_one+0xbf/0x480
	[5147827.460706] [<ffffffff81185f57>] try_to_unmap_anon+0xa7/0x120
	[5147827.466698] [<ffffffff8118601d>] try_to_unmap+0x4d/0x60
	[5147827.472170] [<ffffffff811a8cbe>] migrate_pages+0x43e/0x780
	[5147827.477901] [<ffffffff811a75d0>] ? remove_migration_pte+0x330/0x330
	[5147827.484415] [<ffffffff811a9a34>] migrate_misplaced_page+0xc4/0xf0
	[5147827.490753] [<ffffffff811790ae>] do_numa_page+0x12e/0x1b0
	[5147827.496399] [<ffffffff8117a27f>] handle_mm_fault+0x5ff/0xf10
	[5147827.502304] [<ffffffff8172f7f4>] __do_page_fault+0x184/0x570
	[5147827.508210] [<ffffffff811828d5>] ? change_protection+0x65/0xb0
	[5147827.514291] [<ffffffff810a0535>] ? set_next_entity+0x95/0xb0
	[5147827.520197] [<ffffffff81198c4b>] ? change_prot_numa+0x1b/0x40
	[5147827.526189] [<ffffffff8109f386>] ? task_numa_work+0x266/0x300
	[5147827.532182] [<ffffffff8172fbfa>] do_page_fault+0x1a/0x70
	[5147827.537739] [<ffffffff81013ed7>] ? do_notify_resume+0x97/0xb0
	[5147827.543732] [<ffffffff8172bfc8>] page_fault+0x28/0x30
	[5147827.549029] Code: 08 4c 89 ff 4c 89 23 48 89 4b 08 48 89 19 48 89 55 d0 e8 72 f3 64 00 4c 3b 65 d0 74 23 45 85 ed 75 09 eb 0d 0f 1f 44 00 00 f3 90 <f6> 43 20 01 75 f8 48 83 c4 08 5b 41 5c 41 5d 41 5e 41 5f 5d c3
	[5147831.454840] Code: 5c 24 08 4c 89 ff 4c 89 23 48 89 4b 08 48 89 19 48 89 55 d0 e8 72 f3 64 00 4c 3b 65 d0 74 23 45 85 ed 75 09 eb 0d 0f 1f 44 00 00 <f3> 90 f6 43 20 01 75 f8 48 83 c4 08 5b 41 5c 41 5d 41 5e 41 5f

	[5147831.799104] i7core_edac serio_raw ipmi_si shpchp edac_core acpi_power_meter lpc_ich mac_hid lp parport hid_generic usbhid hid mptsas mptscsih bnx2x mptbase psmouse mdio pata_acpi scsi_transport_sas libcrc32c
	[5147831.818127] CPU: 13 PID: 1209 Comm: diamond Tainted: G W 3.13.0-59-generic #98-Ubuntu
	[5147831.827065] Hardware name: Dell Inc. PowerEdge R610/0F0XJ6, BIOS 6.1.0 10/18/2011
	[5147831.834704] task: ffff880417719800 ti: ffff880422c16000 task.ti: ffff880422c16000
	[5147831.842343] RIP: 0010:[<ffffffff810dc206>] [<ffffffff810dc206>] generic_exec_single+0x86/0xb0
	[5147831.851118] RSP: 0000:ffff880422c179d0 EFLAGS: 00000202
	[5147831.856590] RAX: 0000000000000206 RBX: ffff880422c179f8 RCX: ffff880422189a70
	[5147831.863882] RDX: ffff880418a77a70 RSI: 0000000000000206 RDI: 0000000000000206
	[5147831.871172] RBP: ffff880422c17a00 R08: 0000000000000206 R09: ffff880421c6d020
	[5147831.878462] R10: 000000000000207b R11: 0000000000000000 R12: 0000000000000000
	[5147831.885753] R13: 02ffff0000080008 R14: ffffea000fb48180 R15: 0000000000000000
	[5147831.893043] FS: 00007f4e6c55f740(0000) GS:ffff880227cc0000(0000) knlGS:0000000000000000
	[5147831.901289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
	[5147831.907194] CR2: 00000000008ecb20 CR3: 0000000420f2c000 CR4: 00000000000007e0
	[5147831.914484] Stack:
	[5147831.916661] ffff880418a77a70 000000000000000f 000000000000000d ffffffff81d144c0
	[5147831.924246] ffffffff8105c800 0000000000000301 ffff880422c17a70 ffffffff810dc315
	[5147831.931832] ffffffff8116dd99 0000000000000001 ffff880227cf3c40 ffff880422189a70
	[5147831.939418] Call Trace:
	[5147831.942031] [<ffffffff8105c800>] ? leave_mm+0x80/0x80
	[5147831.947329] [<ffffffff810dc315>] smp_call_function_single+0xe5/0x190
	[5147831.953929] [<ffffffff8116dd99>] ? zone_statistics+0x89/0xa0
	[5147831.959835] [<ffffffff8105c800>] ? leave_mm+0x80/0x80
	[5147831.965135] [<ffffffff810dc746>] smp_call_function_many+0x286/0x2d0
	[5147831.971647] [<ffffffff8105c800>] ? leave_mm+0x80/0x80
	[5147831.976946] [<ffffffff8105c957>] native_flush_tlb_others+0x37/0x40
	[5147831.983372] [<ffffffff8105cc56>] flush_tlb_page+0x56/0xa0
	[5147831.989018] [<ffffffff8118a7d8>] ptep_clear_flush+0x48/0x60
	[5147831.994837] [<ffffffff81184ddf>] try_to_unmap_one+0xbf/0x480
	[5147832.000741] [<ffffffff811a73e5>] ? remove_migration_pte+0x145/0x330
	[5147832.007252] [<ffffffff81185270>] try_to_unmap_file+0xd0/0x630
	[5147832.013245] [<ffffffff811b2aa6>] ? __mem_cgroup_commit_charge+0x156/0x3d0
	[5147832.020276] [<ffffffff81185ffd>] try_to_unmap+0x2d/0x60
	[5147832.025748] [<ffffffff811a8cbe>] migrate_pages+0x43e/0x780
	[5147832.031480] [<ffffffff811a75d0>] ? remove_migration_pte+0x330/0x330
	[5147832.037991] [<ffffffff811a9a34>] migrate_misplaced_page+0xc4/0xf0
	[5147832.044330] [<ffffffff811790ae>] do_numa_page+0x12e/0x1b0
	[5147832.049976] [<ffffffff8117a27f>] handle_mm_fault+0x5ff/0xf10
	[5147832.055881] [<ffffffff8172f7f4>] __do_page_fault+0x184/0x570
	[5147832.061787] [<ffffffff811828d5>] ? change_protection+0x65/0xb0
	[5147832.067865] [<ffffffff810a0535>] ? set_next_entity+0x95/0xb0
	[5147832.073770] [<ffffffff81198c4b>] ? change_prot_numa+0x1b/0x40
	[5147832.079763] [<ffffffff8109f386>] ? task_numa_work+0x266/0x300
	[5147832.085755] [<ffffffff8172fbfa>] do_page_fault+0x1a/0x70
	[5147832.091313] [<ffffffff81013ed7>] ? do_notify_resume+0x97/0xb0
	[5147832.097305] [<ffffffff8172bfc8>] page_fault+0x28/0x30
	[5147832.102603] Code: 4c 89 23 48 89 4b 08 48 89 19 48 89 55 d0 e8 72 f3 64 00 4c 3b 65 d0 74 23 45 85 ed 75 09 eb 0d 0f 1f 44 00 00 f3 90 f6 43 20 01 <75> f8 48 83 c4 08 5b 41 5c 41 5d 41 5e 41 5f 5d c3 44 89 f7 ff
	[5147842.407263] INFO: rcu_sched detected stalls on CPUs/tasks: { 19} (detected by 16, t=330486 jiffies, g=392910865, c=392910864, q=0)
	[5147842.419188] sending NMI to all CPUs:
	[5147842.422928] NMI backtrace for cpu 0

	[5147842.426580] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 3.13.0-59-generic #98-Ubuntu
	[5147842.435345] Hardware name: Dell Inc. PowerEdge R610/0F0XJ6, BIOS 6.1.0 10/18/2011
	[5147842.442982] task: ffffffff81c15480 ti: ffffffff81c00000 task.ti: ffffffff81c00000
	[5147842.450618] RIP: 0010:[<ffffffff8107486d>] [<ffffffff8107486d>] lock_timer_base.isra.35+0x4d/0x50
	[5147842.459742] RSP: 0018:ffff88042fc03978 EFLAGS: 00000246
	[5147842.465213] RAX: 000000000000584d RBX: 0000000000000000 RCX: 0000000000007530
	[5147842.472504] RDX: 0000000000000000 RSI: ffff88042fc039a8 RDI: ffff8800b82aa8d8
	[5147842.479794] RBP: ffff88042fc03998 R08: 0000000000000001 R09: 000000000000000c
	[5147842.487083] R10: 0000000000000000 R11: 0000000000000007 R12: 0000000000000000
	[5147842.494375] R13: ffff8800b82aa8d8 R14: ffff88042fc039a8 R15: 000000014caeab3a
	[5147842.501666] FS: 0000000000000000(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000
	[5147842.509910] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
	[5147842.515815] CR2: 00007ff62626fc68 CR3: 0000000001c0e000 CR4: 00000000000007f0
	[5147842.523107] Stack:
	[5147842.525284] ffff8800b82aa838 ffff8800b82aa8c0 ffff880202d2f600 0000000000007530
	[5147842.532870] ffff88042fc039d8 ffffffff81075ee4 ffffffff7540e038 ffff8800b82aa838
	[5147842.540457] 0000000000000000 ffff880202d2f600 0000000000007530 0000000000000007
	[5147842.548044] Call Trace:
	[5147842.550655] <IRQ>
	[5147842.552745] [<ffffffff81075ee4>] mod_timer_pending+0x44/0x180
	[5147842.558932] [<ffffffffa02c18f0>] __nf_ct_refresh_acct+0xb0/0xc0 [nf_conntrack]
	[5147842.566400] [<ffffffffa02c8caf>] tcp_packet+0x47f/0x8d0 [nf_conntrack]
	[5147842.573173] [<ffffffffa02dc5c3>] ? __br_deliver+0x53/0x120 [bridge]
	[5147842.579687] [<ffffffffa02c25f1>] ? __nf_conntrack_find_get+0x31/0x160 [nf_conntrack]
	[5147842.587673] [<ffffffffa02c3574>] nf_conntrack_in+0x394/0xb60 [nf_conntrack]
	[5147842.594880] [<ffffffff8165d0b0>] ? inet_del_offload+0x40/0x40
	[5147842.600874] [<ffffffffa01f62e2>] ipv4_conntrack_in+0x22/0x30 [nf_conntrack_ipv4]
	[5147842.608513] [<ffffffff81656b9a>] nf_iterate+0x9a/0xb0
	[5147842.613812] [<ffffffff8165d0b0>] ? inet_del_offload+0x40/0x40
	[5147842.619804] [<ffffffff81656c24>] nf_hook_slow+0x74/0x130
	[5147842.625361] [<ffffffff8165d0b0>] ? inet_del_offload+0x40/0x40
	[5147842.631354] [<ffffffff8165dad8>] ip_rcv+0x2f8/0x3d0
	[5147842.636480] [<ffffffff81627276>] __netif_receive_skb_core+0x666/0x840
	[5147842.643166] [<ffffffff81627468>] __netif_receive_skb+0x18/0x60
	[5147842.649244] [<ffffffff816274d3>] netif_receive_skb+0x23/0x90
	[5147842.655150] [<ffffffff81627f20>] napi_gro_receive+0x80/0xb0
	[5147842.660978] [<ffffffffa00b6fc1>] bnx2x_rx_int+0x1021/0x18a0 [bnx2x]
	[5147842.667489] [<ffffffff816157b7>] ? kfree_skbmem+0x37/0xa0
	[5147842.673136] [<ffffffff816263c5>] ? dev_kfree_skb_any+0x35/0x40
	[5147842.679215] [<ffffffff810cda93>] ? ktime_get+0x43/0xc0
	[5147842.684608] [<ffffffffa00b793a>] bnx2x_poll+0xfa/0x3e0 [bnx2x]
	[5147842.690687] [<ffffffff81627852>] net_rx_action+0x152/0x250
	[5147842.696421] [<ffffffff8106cccc>] __do_softirq+0xec/0x2c0
	[5147842.701981] [<ffffffff8106d215>] irq_exit+0x105/0x110
	[5147842.707279] [<ffffffff81736536>] do_IRQ+0x56/0xc0
	[5147842.712231] [<ffffffff8172bbee>] common_interrupt+0x6e/0x6e
	[5147842.718049] <EOI>
	[5147842.720139] [<ffffffff815d572f>] ? cpuidle_enter_state+0x4f/0xc0
	[5147842.726585] [<ffffffff815d5859>] cpuidle_idle_call+0xb9/0x1f0
	[5147842.732577] [<ffffffff8101d3ee>] arch_cpu_idle+0xe/0x30
	[5147842.738049] [<ffffffff810bf215>] cpu_startup_entry+0xc5/0x290
	[5147842.744040] [<ffffffff81711907>] rest_init+0x77/0x80
	[5147842.749252] [<ffffffff81d34f70>] start_kernel+0x438/0x443
	[5147842.754898] [<ffffffff81d34941>] ? repair_env_string+0x5c/0x5c
	[5147842.760977] [<ffffffff81d34120>] ? early_idt_handlers+0x120/0x120
	[5147842.767317] [<ffffffff81d345ee>] x86_64_start_reservations+0x2a/0x2c
	[5147842.773915] [<ffffffff81d34733>] x86_64_start_kernel+0x143/0x152
	[5147842.780166] Code: 28 4c 89 e7 e8 05 6f 6b 00 49 89 06 49 3b 5d 00 75 0c 5b 4c 89 e0 41 5c 41 5d 41 5e 5d c3 48 89 c6 4c 89 e7 e8 f5 6c 6b 00 f3 90 <eb> c7 90 55 48 89 e5 41 57 49 89 ff 41 56 41 55 49 89 d5 41 5

Event Timeline

akosiaris edited the content of this paste. (Show Details)Sep 25 2015, 11:24 AM

akosiaris changed the title of this paste from untitled to labnet1002 kernel stack trace .

akosiaris updated the paste's language from autodetect to autodetect.

akosiaris added a project: acl*sre-team.

akosiaris edited the content of this paste. (Show Details)Sep 25 2015, 11:27 AM

labnet1002 kernel stack trace ActivePublicActions

Event Timeline

labnet1002 kernel stack trace
ActivePublic
Actions