Page MenuHomePhabricator

analytics1061 is down
Closed, ResolvedPublic

Description

Host is unreachable, console says:

[20713362.167904] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [container-execu:16589]
[20713362.177334] Modules linked in: cpuid nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_NFLOG xt_limit xt_tcpudp xt_pkttype nf_conntrack_ipv6 nf_defrag_ipv6 binfmt_misc xt_conntrack ip6table_filter ip6_tables nfnetlink_log nfnetlink intel_rapl sb_edac edac_core x86_pkg_temp_thermal intel_powerclamp coretemp kvm mgag200 irqbypass ttm drm_kms_helper drm iTCO_wdt crct10dif_pclmul crc32_pclmul[20713362.215905] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [DataXceiver for:16760]
[20713362.215905] Modules linked in:
[20713362.215906]  cpuid nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_NFLOG xt_limit xt_tcpudp xt_pkttype nf_conntrack_ipv6 nf_defrag_ipv6 binfmt_misc xt_conntrack ip6table_filter ip6_tables nfnetlink_log nfnetlink intel_rapl sb_edac edac_core x86_pkg_temp_thermal intel_powerclamp coretemp kvm mgag200 irqbypass ttm drm_kms_helper drm iTCO_wdt crct10dif_pclmul crc32_pclmul i2c_algo_bit sg iTCO_vendor_support evdev mxm_wmi ghash_clmulni_intel mei_me dcdbas lpc_ich pcspkr mei shpchp mfd_core wmi ipmi_si button ipmi_devintf ipmi_msghandler nf_conntrack ip_tables x_tables autofs4 ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache dm_mod sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper ahci lrw gf128mul ehci_pci ablk_helper libahci ehci_hcd tg3 cryptd ptp libata megaraid_sas usbcore pps_core usb_common libphy scsi_mod[20713362.215954] CPU: 16 PID: 16760 Comm: DataXceiver for Tainted: G      D      L  4.9.0-9-amd64 #1 Debian 4.9.168-1+deb9u2
[20713362.215955] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.3.4 11/08/2016
[20713362.215956] task: ffff9f79b0c5d100 task.stack: ffffb32864b6c000
[20713362.215957] RIP: 0010:[<ffffffffb9ac5bdf>]
[20713362.215960]  [<ffffffffb9ac5bdf>] native_queued_spin_lock_slowpath+0x10f/0x1a0
[20713362.215961] RSP: 0018:ffffb32864b6fc98  EFLAGS: 00000246
[20713362.215962] RAX: 0000000000000000 RBX: ffff9f80d6fbae00 RCX: 0000000000000008
[20713362.215963] RDX: ffff9f80dfc19480 RSI: 0000000000440000 RDI: ffff9f80d6fbae58
[20713362.215964] RBP: ffff9f7a666661a8 R08: 0000000000000000 R09: 0000000000000000
[20713362.215964] R10: ffff9f88db5ba800 R11: ffffeedc41592840 R12: ffff9f7a66666230
[20713362.215965] R13: ffffffffba402e90 R14: ffff9f80d6fbae58 R15: 0000000000000000
[20713362.215967] FS:  00007f10a0c05700(0000) GS:ffff9f80dfc00000(0000) knlGS:0000000000000000
[20713362.215968] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[20713362.215968] CR2: 00000000e2bd1000 CR3: 00000002119c8000 CR4: 0000000000160670
[20713362.215969] Stack:
[20713362.215970]  ffffffffba01a79d
[20713362.215971]  ffffffffb9c3ab3a ffff9f7a666661a8 0000000000000000[20713362.215972]  0000000000000001
[20713362.215972]  ffff9f7a666661a8 ffff9f7a66666230 0000000000000000[20713362.215974]  ffffffffb9c3b10d
[20713362.215974]  ffff9f7a666661a8 ffffb32864b6fd40 0000000000000000[20713362.215975] Call Trace:
[20713362.215979]  [<ffffffffba01a79d>] ? _raw_spin_lock+0x1d/0x20
[20713362.215983]  [<ffffffffb9c3ab3a>] ? locked_inode_to_wb_and_lock_list+0x5a/0x180
[20713362.215985]  [<ffffffffb9c3b10d>] ? __mark_inode_dirty+0x24d/0x360
[20713362.215987]  [<ffffffffb9c284d9>] ? generic_update_time+0x79/0xd0
[20713362.215988]  [<ffffffffb9c286e6>] ? current_time+0x36/0x70
[20713362.215989]  [<ffffffffb9c287df>] ? file_update_time+0xbf/0x110
[20713362.215992]  [<ffffffffb9b84ef9>] ? __generic_file_write_iter+0x99/0x1b0
[20713362.216017]  [<ffffffffc02d4200>] ? ext4_file_write_iter+0x90/0x380 [ext4]
[20713362.216019]  [<ffffffffb9ef3a07>] ? sock_write_iter+0x87/0x100
[20713362.216022]  [<ffffffffb9c0b450>] ? new_sync_write+0xe0/0x130
[20713362.216023]  [<ffffffffb9c0bc40>] ? vfs_write+0xb0/0x190
[20713362.216025]  [<ffffffffb9c0d082>] ? SyS_write+0x52/0xc0
[20713362.216027]  [<ffffffffb9a03b7d>] ? do_syscall_64+0x8d/0x100
[20713362.216029]  [<ffffffffba01a88e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
[20713362.216029] Code:
[20713362.216030] 12 83 e0 03 83 e9 01 48 c1 e0 04 48 63 c9 48 05 80 94 01 00 48 03 04 cd e0 f3 46 ba 48 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 <85> c0 74 f7 4c 8b 02 4d 85 c0 74 08 41 0f 18 08 eb 02 f3 90 8b [20713362.239905] NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [java:14604]
[20713362.239905] Modules linked in:
[20713362.239906]  cpuid nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_NFLOG xt_limit xt_tcpudp xt_pkttype nf_conntrack_ipv6 nf_defrag_ipv6 binfmt_misc xt_conntrack ip6table_filter ip6_tables nfnetlink_log nfnetlink intel_rapl sb_edac edac_core x86_pkg_temp_thermal intel_powerclamp coretemp kvm mgag200 irqbypass ttm drm_kms_helper drm iTCO_wdt crct10dif_pclmul crc32_pclmul i2c_algo_bit sg iTCO_vendor_support evdev mxm_wmi ghash_clmulni_intel mei_me dcdbas lpc_ich pcspkr mei shpchp mfd_core wmi ipmi_si button ipmi_devintf ipmi_msghandler nf_conntrack ip_tables x_tables autofs4 ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache dm_mod sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper ahci lrw gf128mul ehci_pci ablk_helper libahci ehci_hcd tg3 cryptd ptp libata megaraid_sas usbcore pps_core usb_common libphy scsi_mod[20713362.239948] CPU: 22 PID: 14604 Comm: java Tainted: G      D      L  4.9.0-9-amd64 #1 Debian 4.9.168-1+deb9u2
[20713362.239949] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.3.4 11/08/2016
[20713362.239950] task: ffff9f85f0cfc0c0 task.stack: ffffb32862ff4000
[20713362.239950] RIP: 0010:[<ffffffffb9ac5b2f>]
[20713362.239953]  [<ffffffffb9ac5b2f>] native_queued_spin_lock_slowpath+0x5f/0x1a0
[20713362.239954] RSP: 0018:ffffb32862ff7e40  EFLAGS: 00000202
[20713362.239954] RAX: 0000000000440101 RBX: ffff9f80d6fbae00 RCX: 0000000000000001
[20713362.239955] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f80d6fbae58
[20713362.239956] RBP: ffff9f864726b228 R08: 0000000000000000 R09: 0000000000000000
[20713362.239957] R10: ffff9f7aba598d50 R11: ffffeedc4a0ec280 R12: ffff9f864726b2b0
[20713362.239958] R13: ffffffffba402e90 R14: ffff9f80d6fbae58 R15: 00000000ffffff9c
[20713362.239959] FS:  00007f86c6a03700(0000) GS:ffff9f80dfcc0000(0000) knlGS:0000000000000000
[20713362.239960] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[20713362.239961] CR2: 00000000e8a7e010 CR3: 0000000941a9a000 CR4: 0000000000160670
[20713362.239961] Stack:
[20713362.239962]  ffffffffba01a79d
[20713362.239963]  ffffffffb9c3ab3a ffff9f81acdb0300 ffff9f864726b228[20713362.239964]  ffff9f864726b228
[20713362.239964]  ffffffffc032b5a0 0000000000000000 ffff9f864726b228[20713362.239965]  ffffffffb9c3c5ce
[20713362.239966]  ffff9f864726b228 ffff9f7d8d421000 ffffffffb9c28d30[20713362.239967] Call Trace:
[20713362.239969]  [<ffffffffba01a79d>] ? _raw_spin_lock+0x1d/0x20
[20713362.239972]  [<ffffffffb9c3ab3a>] ? locked_inode_to_wb_and_lock_list+0x5a/0x180
[20713362.239975]  [<ffffffffb9c3c5ce>] ? inode_io_list_del+0x1e/0x40
[20713362.239976]  [<ffffffffb9c28d30>] ? evict+0x50/0x180
[20713362.239978]  [<ffffffffb9c1cb28>] ? do_unlinkat+0x148/0x330
[20713362.239980]  [<ffffffffb9a03b7d>] ? do_syscall_64+0x8d/0x100
[20713362.239982]  [<ffffffffba01a88e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
[20713362.239983] Code:
[20713362.239983] 75 42 f0 0f ba 2f 08 b8 00 01 00 00 0f 42 f0 8b 07 30 e4 09 c6 f7 c6 00 ff ff ff 75 1b 85 f6 74 0e 8b 07 84 c0 74 08 f3 90 8b 07 <84> c0 75 f8 b8 01 00 00 00 66 89 07 c3 81 e6 00 ff 00 00 75 04  i2c_algo_bit sg iTCO_vendor_support evdev mxm_wmi ghash_clmulni_intel mei_me dcdbas lpc_ich pcspkr mei shpchp mfd_core wmi ipmi_si button ipmi_devintf ipmi_msghandler nf_conntrack ip_tables x_tables autofs4 ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache dm_mod sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper ahci lrw gf128mul ehci_pci ablk_helper libahci ehci_hcd tg3 cryptd ptp libata megaraid_sas usbcore pps_core usb_common libphy scsi_mod
[20713362.942999] CPU: 8 PID: 16589 Comm: container-execu Tainted: G      D      L  4.9.0-9-amd64 #1 Debian 4.9.168-1+deb9u2
[20713362.955226] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.3.4 11/08/2016
[20713362.964060] task: ffff9f79fc072080 task.stack: ffffb32864a64000
[20713362.970954] RIP: 0010:[<ffffffffb9ac5bf5>]  [<ffffffffb9ac5bf5>] native_queued_spin_lock_slowpath+0x125/0x1a0
[20713362.982320] RSP: 0018:ffffb32864a67b18  EFLAGS: 00000202
[20713362.988536] RAX: 0000000000000000 RBX: ffff9f80d6fbae00 RCX: 0000000000440101
[20713362.996788] RDX: ffff9f80dfb19480 RSI: 0000000000240000 RDI: ffff9f80d6fbae58
[20713363.005039] RBP: ffff9f7a8a8bd698 R08: 0000000000000000 R09: 0000000000000000
[20713363.013291] R10: ffff9f88db5ba800 R11: ffffeedc804e0440 R12: ffff9f7a8a8bd720
[20713363.021543] R13: ffffffffba402e90 R14: ffff9f80d6fbae58 R15: 0000000000000000
[20713363.029795] FS:  00007f20dd8ac440(0000) GS:ffff9f80dfb00000(0000) knlGS:0000000000000000
[20713363.039114] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[20713363.045816] CR2: 00007ffee785dff8 CR3: 0000000ba5d8c000 CR4: 0000000000160670
[20713363.054067] Stack:
[20713363.056597]  ffffffffba01a79d ffffffffb9c3ab3a ffffb32864a67cb8 0000000000000000
[20713363.065173]  0000000000000003 ffff9f7a8a8bd698 ffff9f7a8a8bd720 0000000000000000
[20713363.073750]  ffffffffb9c3b10d ffff9f88db695800 ffff9f7a8a8bd698 ffffb32864a67c94
[20713363.082326] Call Trace:
[20713363.085343]  [<ffffffffba01a79d>] ? _raw_spin_lock+0x1d/0x20
[20713363.091949]  [<ffffffffb9c3ab3a>] ? locked_inode_to_wb_and_lock_list+0x5a/0x180
[20713363.100397]  [<ffffffffb9c3b10d>] ? __mark_inode_dirty+0x24d/0x360
[20713363.107602]  [<ffffffffc0317b48>] ? ext4_mb_new_blocks+0xe8/0xaf0 [ext4]
[20713363.115378]  [<ffffffffc03080c4>] ? ext4_find_extent+0x264/0x310 [ext4]
[20713363.123056]  [<ffffffffc030cca2>] ? ext4_ext_map_blocks+0xb82/0x1320 [ext4]
[20713363.131121]  [<ffffffffc030fd05>] ? __ext4_handle_dirty_metadata+0x45/0x1c0 [ext4]
[20713363.139865]  [<ffffffffc00b8d2f>] ? do_get_write_access+0x33f/0x450 [jbd2]
[20713363.147834]  [<ffffffffc02da964>] ? ext4_map_blocks+0x164/0x5d0 [ext4]
[20713363.155415]  [<ffffffffc02db860>] ? ext4_getblk+0x50/0x190 [ext4]
[20713363.162510]  [<ffffffffc02db9bf>] ? ext4_bread+0x1f/0xb0 [ext4]
[20713363.169411]  [<ffffffffc02e54e8>] ? ext4_append+0x48/0xd0 [ext4]
[20713363.176410]  [<ffffffffc02ea956>] ? ext4_mkdir+0x276/0x470 [ext4]
[20713363.183500]  [<ffffffffb9c18e0c>] ? vfs_mkdir+0x10c/0x1a0
[20713363.189814]  [<ffffffffb9c1d7fe>] ? SyS_mkdir+0xce/0x120
[20713363.196032]  [<ffffffffb9a03b7d>] ? do_syscall_64+0x8d/0x100
[20713363.202637]  [<ffffffffba01a88e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
[20713363.210695] Code: 04 cd e0 f3 46 ba 48 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 85 c0 74 f7 4c 8b 02 4d 85 c0 74 08 41 0f 18 08 eb 02 f3 90 8b 0f <66> 85 c9 75 f7 89 c8 66 31 c0 39 c6 74 50 4d 85 c0 c6 07 01 74
[20713366.176025] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [kworker/u50:8:20807]
[20713366.185260] Modules linked in: cpuid nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_NFLOG xt_limit xt_tcpudp xt_pkttype nf_conntrack_ipv6 nf_defrag_ipv6 binfmt_misc xt_conntrack ip6table_filter ip6_tables nfnetlink_log nfnetlink intel_rapl sb_edac edac_core x86_pkg_temp_thermal intel_powerclamp coretemp kvm mgag200 irqbypass ttm drm_kms_helper drm iTCO_wdt crct10dif_pclmul crc32_pclmul i2c_algo_bit sg iTCO_vendor_support evdev mxm_wmi ghash_clmulni_intel mei_me dcdbas lpc_ich pcspkr mei shpchp mfd_core wmi ipmi_si button ipmi_devintf ipmi_msghandler nf_conntrack ip_tables x_tables autofs4 ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache dm_mod sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper ahci lrw gf128mul ehci_pci ablk_helper libahci ehci_hcd tg3 cryptd ptp libata megaraid_sas usbcore pps_core usb_common libphy scsi_mod
[20713366.269065] CPU: 9 PID: 20807 Comm: kworker/u50:8 Tainted: G      D      L  4.9.0-9-amd64 #1 Debian 4.9.168-1+deb9u2
[20713366.281098] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.3.4 11/08/2016
[20713366.289938] Workqueue: writeback wb_workfn (flush-8:32)
[20713366.296078] task: ffff9f84363ab140 task.stack: ffffb3284c894000
[20713366.302974] RIP: 0010:[<ffffffffb9ac5b2f>]  [<ffffffffb9ac5b2f>] native_queued_spin_lock_slowpath+0x5f/0x1a0
[20713366.314245] RSP: 0018:ffffb3284c897bc8  EFLAGS: 00000202
[20713366.320461] RAX: 0000000000000101 RBX: ffff9f80d6fbae28 RCX: ffff9f7b22ad7a88
[20713366.328713] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f7b22ad7b10
[20713366.336967] RBP: ffff9f88db78a800 R08: ffffb3284c897c78 R09: 0000000000ffffff
[20713366.345219] R10: 00000000f0f0f0f0 R11: 00000000fa83b2da R12: ffff9f7b22ad7b78
[20713366.353471] R13: 0000000000000000 R14: ffffb3284c897db0 R15: ffff9f80d6fbae00
[20713366.361725] FS:  0000000000000000(0000) GS:ffff9f88df300000(0000) knlGS:0000000000000000
[20713366.371045] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[20713366.377746] CR2: 00007f849435fa58 CR3: 00000005ddc08000 CR4: 0000000000160670
[20713366.385999] Stack:
[20713366.388531]  ffffffffba01a79d ffffffffb9c3b94e ffff9f80d6fbae58 ffff9f84363ab140
[20713366.397112]  ffff9f80d6fbae38 00000002347bda99 ffff9f7b22ad7b10 ffff9f7b22ad7a88
[20713366.405694]  ffff9f80d6f5e9c0 0000000000000000 0000000000000000 0000000000000000
[20713366.414277] Call Trace:
[20713366.417296]  [<ffffffffba01a79d>] ? _raw_spin_lock+0x1d/0x20
[20713366.423902]  [<ffffffffb9c3b94e>] ? writeback_sb_inodes+0x13e/0x4f0
[20713366.431187]  [<ffffffffb9c3bd87>] ? __writeback_inodes_wb+0x87/0xb0
[20713366.438471]  [<ffffffffb9c3c0fe>] ? wb_writeback+0x27e/0x310
[20713366.445078]  [<ffffffffb9c2798c>] ? get_nr_inodes+0x3c/0x60
[20713366.451587]  [<ffffffffb9c3ca64>] ? wb_workfn+0x2b4/0x380
[20713366.457902]  [<ffffffffb9a9460a>] ? process_one_work+0x18a/0x430
[20713366.464894]  [<ffffffffb9a948fd>] ? worker_thread+0x4d/0x490
[20713366.471498]  [<ffffffffb9a948b0>] ? process_one_work+0x430/0x430
[20713366.478491]  [<ffffffffb9a9a969>] ? kthread+0xd9/0xf0
[20713366.484416]  [<ffffffffba01a9a4>] ? __switch_to_asm+0x34/0x70
[20713366.491117]  [<ffffffffb9a9a890>] ? kthread_park+0x60/0x60
[20713366.497527]  [<ffffffffba01aa37>] ? ret_from_fork+0x57/0x70
[20713366.504034] Code: 75 42 f0 0f ba 2f 08 b8 00 01 00 00 0f 42 f0 8b 07 30 e4 09 c6 f7 c6 00 ff ff ff 75 1b 85 f6 74 0e 8b 07 84 c0 74 08 f3 90 8b 07 <84> c0 75 f8 b8 01 00 00 00 66 89 07 c3 81 e6 00 ff 00 00 75 04
[20713390.168740] NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [container-execu:16589]
[20713390.178170] Modules linked in: cpuid nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_NFLOG xt_limit xt_tcpudp xt_pkttype nf_conntrack_ipv6 nf_defrag_ipv6 binfmt_misc xt_conntrack ip6table_filter ip6_tables nfnetlink_log nfnetlink intel_rapl sb_edac edac_core x86_pkg_temp_thermal intel_powerclamp coretemp kvm mgag200 irqbypass ttm drm_kms_helper drm iTCO_wdt crct10dif_pclmul crc32_pclmul[20713390.216742] NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [DataXceiver for:16760]
[20713390.216742] Modules linked in:
[20713390.216743]  cpuid nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_NFLOG xt_limit xt_tcpudp xt_pkttype nf_conntrack_ipv6 nf_defrag_ipv6 binfmt_misc xt_conntrack ip6table_filter ip6_tables nfnetlink_log nfnetlink intel_rapl sb_edac edac_core x86_pkg_temp_thermal intel_powerclamp coretemp kvm mgag200 irqbypass ttm drm_kms_helper drm iTCO_wdt crct10dif_pclmul crc32_pclmul i2c_algo_bit sg iTCO_vendor_support evdev mxm_wmi ghash_clmulni_intel mei_me dcdbas lpc_ich pcspkr mei shpchp mfd_core wmi ipmi_si button ipmi_devintf ipmi_msghandler nf_conntrack ip_tables x_tables autofs4 ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache dm_mod sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper ahci lrw gf128mul ehci_pci ablk_helper libahci ehci_hcd tg3 cryptd ptp libata megaraid_sas usbcore pps_core usb_common libphy scsi_mod[20713390.216792] CPU: 16 PID: 16760 Comm: DataXceiver for Tainted: G      D      L  4.9.0-9-amd64 #1 Debian 4.9.168-1+deb9u2
[20713390.216793] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.3.4 11/08/2016
[20713390.216793] task: ffff9f79b0c5d100 task.stack: ffffb32864b6c000
[20713390.216794] RIP: 0010:[<ffffffffb9ac5bdc>]
[20713390.216797]  [<ffffffffb9ac5bdc>] native_queued_spin_lock_slowpath+0x10c/0x1a0
[20713390.216798] RSP: 0018:ffffb32864b6fc98  EFLAGS: 00000246
[20713390.216799] RAX: 0000000000000000 RBX: ffff9f80d6fbae00 RCX: 0000000000000008
[20713390.216800] RDX: ffff9f80dfc19480 RSI: 0000000000440000 RDI: ffff9f80d6fbae58
[20713390.216801] RBP: ffff9f7a666661a8 R08: 0000000000000000 R09: 0000000000000000
[20713390.216801] R10: ffff9f88db5ba800 R11: ffffeedc41592840 R12: ffff9f7a66666230
[20713390.216802] R13: ffffffffba402e90 R14: ffff9f80d6fbae58 R15: 0000000000000000
[20713390.216804] FS:  00007f10a0c05700(0000) GS:ffff9f80dfc00000(0000) knlGS:0000000000000000
[20713390.216804] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[20713390.216805] CR2: 00000000e2bd1000 CR3: 00000002119c8000 CR4: 0000000000160670
[20713390.216806] Stack:
[20713390.216807]  ffffffffba01a79d
[20713390.216808]  ffffffffb9c3ab3a ffff9f7a666661a8 0000000000000000[20713390.216809]  0000000000000001
[20713390.216809]  ffff9f7a666661a8 ffff9f7a66666230 0000000000000000[20713390.216811]  ffffffffb9c3b10d
[20713390.216811]  ffff9f7a666661a8 ffffb32864b6fd40 0000000000000000[20713390.216812] Call Trace:
[20713390.216816]  [<ffffffffba01a79d>] ? _raw_spin_lock+0x1d/0x20
[20713390.216820]  [<ffffffffb9c3ab3a>] ? locked_inode_to_wb_and_lock_list+0x5a/0x180
[20713390.216822]  [<ffffffffb9c3b10d>] ? __mark_inode_dirty+0x24d/0x360
[20713390.216824]  [<ffffffffb9c284d9>] ? generic_update_time+0x79/0xd0
[20713390.216825]  [<ffffffffb9c286e6>] ? current_time+0x36/0x70
[20713390.216826]  [<ffffffffb9c287df>] ? file_update_time+0xbf/0x110
[20713390.216829]  [<ffffffffb9b84ef9>] ? __generic_file_write_iter+0x99/0x1b0
[20713390.216853]  [<ffffffffc02d4200>] ? ext4_file_write_iter+0x90/0x380 [ext4]
[20713390.216856]  [<ffffffffb9ef3a07>] ? sock_write_iter+0x87/0x100
[20713390.216859]  [<ffffffffb9c0b450>] ? new_sync_write+0xe0/0x130
[20713390.216861]  [<ffffffffb9c0bc40>] ? vfs_write+0xb0/0x190
[20713390.216863]  [<ffffffffb9c0d082>] ? SyS_write+0x52/0xc0
[20713390.216864]  [<ffffffffb9a03b7d>] ? do_syscall_64+0x8d/0x100
[20713390.216866]  [<ffffffffba01a88e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
[20713390.216867] Code:
[20713390.216867] 41 c1 e9 12 83 e0 03 83 e9 01 48 c1 e0 04 48 63 c9 48 05 80 94 01 00 48 03 04 cd e0 f3 46 ba 48 89 10 8b 42 08 85 c0 75 09 f3 90 <8b> 42 08 85 c0 74 f7 4c 8b 02 4d 85 c0 74 08 41 0f 18 08 eb 02 [20713390.240742] NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [java:14604]
[20713390.240742] Modules linked in:
[20713390.240743]  cpuid nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_NFLOG xt_limit xt_tcpudp xt_pkttype nf_conntrack_ipv6 nf_defrag_ipv6 binfmt_misc xt_conntrack ip6table_filter ip6_tables nfnetlink_log nfnetlink intel_rapl sb_edac edac_core x86_pkg_temp_thermal intel_powerclamp coretemp kvm mgag200 irqbypass ttm drm_kms_helper drm iTCO_wdt crct10dif_pclmul crc32_pclmul i2c_algo_bit sg iTCO_vendor_support evdev mxm_wmi ghash_clmulni_intel mei_me dcdbas lpc_ich pcspkr mei shpchp mfd_core wmi ipmi_si button ipmi_devintf ipmi_msghandler nf_conntrack ip_tables x_tables autofs4 ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache dm_mod sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper ahci lrw gf128mul ehci_pci ablk_helper libahci ehci_hcd tg3 cryptd ptp libata megaraid_sas usbcore pps_core usb_common libphy scsi_mod[20713390.240785] CPU: 22 PID: 14604 Comm: java Tainted: G      D      L  4.9.0-9-amd64 #1 Debian 4.9.168-1+deb9u2
[20713390.240786] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.3.4 11/08/2016
[20713390.240787] task: ffff9f85f0cfc0c0 task.stack: ffffb32862ff4000
[20713390.240787] RIP: 0010:[<ffffffffb9ac5b2d>]
[20713390.240790]  [<ffffffffb9ac5b2d>] native_queued_spin_lock_slowpath+0x5d/0x1a0
[20713390.240791] RSP: 0018:ffffb32862ff7e40  EFLAGS: 00000202
[20713390.240792] RAX: 0000000000440101 RBX: ffff9f80d6fbae00 RCX: 0000000000000001
[20713390.240792] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f80d6fbae58
[20713390.240793] RBP: ffff9f864726b228 R08: 0000000000000000 R09: 0000000000000000
[20713390.240794] R10: ffff9f7aba598d50 R11: ffffeedc4a0ec280 R12: ffff9f864726b2b0
[20713390.240795] R13: ffffffffba402e90 R14: ffff9f80d6fbae58 R15: 00000000ffffff9c
[20713390.240796] FS:  00007f86c6a03700(0000) GS:ffff9f80dfcc0000(0000) knlGS:0000000000000000
[20713390.240797] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[20713390.240798] CR2: 00000000e8a7e010 CR3: 0000000941a9a000 CR4: 0000000000160670
[20713390.240799] Stack:
[20713390.240799]  ffffffffba01a79d
[20713390.240800]  ffffffffb9c3ab3a ffff9f81acdb0300 ffff9f864726b228[20713390.240801]  ffff9f864726b228
[20713390.240802]  ffffffffc032b5a0 0000000000000000 ffff9f864726b228[20713390.240803]  ffffffffb9c3c5ce
[20713390.240803]  ffff9f864726b228 ffff9f7d8d421000 ffffffffb9c28d30[20713390.240804] Call Trace:
[20713390.240807]  [<ffffffffba01a79d>] ? _raw_spin_lock+0x1d/0x20
[20713390.240810]  [<ffffffffb9c3ab3a>] ? locked_inode_to_wb_and_lock_list+0x5a/0x180
[20713390.240812]  [<ffffffffb9c3c5ce>] ? inode_io_list_del+0x1e/0x40
[20713390.240814]  [<ffffffffb9c28d30>] ? evict+0x50/0x180
[20713390.240815]  [<ffffffffb9c1cb28>] ? do_unlinkat+0x148/0x330
[20713390.240817]  [<ffffffffb9a03b7d>] ? do_syscall_64+0x8d/0x100
[20713390.240819]  [<ffffffffba01a88e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
[20713390.240820] Code:
[20713390.240820] 85 f6 75 42 f0 0f ba 2f 08 b8 00 01 00 00 0f 42 f0 8b 07 30 e4 09 c6 f7 c6 00 ff ff ff 75 1b 85 f6 74 0e 8b 07 84 c0 74 08 f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 c3 81 e6 00 ff 00 00  i2c_algo_bit sg iTCO_vendor_support evdev mxm_wmi ghash_clmulni_intel mei_me dcdbas lpc_ich pcspkr mei shpchp mfd_core wmi ipmi_si button ipmi_devintf ipmi_msghandler nf_conntrack ip_tables x_tables autofs4 ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache dm_mod sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper ahci lrw gf128mul ehci_pci ablk_helper libahci ehci_hcd tg3 cryptd ptp libata megaraid_sas usbcore pps_core usb_common libphy scsi_mod
[20713390.943853] CPU: 8 PID: 16589 Comm: container-execu Tainted: G      D      L  4.9.0-9-amd64 #1 Debian 4.9.168-1+deb9u2
[20713390.956082] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.3.4 11/08/2016
[20713390.964916] task: ffff9f79fc072080 task.stack: ffffb32864a64000
[20713390.971810] RIP: 0010:[<ffffffffb9ac5bf1>]  [<ffffffffb9ac5bf1>] native_queued_spin_lock_slowpath+0x121/0x1a0
[20713390.983176] RSP: 0018:ffffb32864a67b18  EFLAGS: 00000202
[20713390.989392] RAX: 0000000000000000 RBX: ffff9f80d6fbae00 RCX: 0000000000440101
[20713390.997644] RDX: ffff9f80dfb19480 RSI: 0000000000240000 RDI: ffff9f80d6fbae58
[20713391.005895] RBP: ffff9f7a8a8bd698 R08: 0000000000000000 R09: 0000000000000000
[20713391.014147] R10: ffff9f88db5ba800 R11: ffffeedc804e0440 R12: ffff9f7a8a8bd720
[20713391.022399] R13: ffffffffba402e90 R14: ffff9f80d6fbae58 R15: 0000000000000000
[20713391.030651] FS:  00007f20dd8ac440(0000) GS:ffff9f80dfb00000(0000) knlGS:0000000000000000
[20713391.039970] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[20713391.046671] CR2: 00007ffee785dff8 CR3: 0000000ba5d8c000 CR4: 0000000000160670
[20713391.054922] Stack:
[20713391.057453]  ffffffffba01a79d ffffffffb9c3ab3a ffffb32864a67cb8 0000000000000000
[20713391.066029]  0000000000000003 ffff9f7a8a8bd698 ffff9f7a8a8bd720 0000000000000000
[20713391.074606]  ffffffffb9c3b10d ffff9f88db695800 ffff9f7a8a8bd698 ffffb32864a67c94
[20713391.083182] Call Trace:
[20713391.086199]  [<ffffffffba01a79d>] ? _raw_spin_lock+0x1d/0x20
[20713391.092805]  [<ffffffffb9c3ab3a>] ? locked_inode_to_wb_and_lock_list+0x5a/0x180
[20713391.101253]  [<ffffffffb9c3b10d>] ? __mark_inode_dirty+0x24d/0x360
[20713391.108457]  [<ffffffffc0317b48>] ? ext4_mb_new_blocks+0xe8/0xaf0 [ext4]
[20713391.116234]  [<ffffffffc03080c4>] ? ext4_find_extent+0x264/0x310 [ext4]
[20713391.123912]  [<ffffffffc030cca2>] ? ext4_ext_map_blocks+0xb82/0x1320 [ext4]
[20713391.131980]  [<ffffffffc030fd05>] ? __ext4_handle_dirty_metadata+0x45/0x1c0 [ext4]
[20713391.140723]  [<ffffffffc00b8d2f>] ? do_get_write_access+0x33f/0x450 [jbd2]
[20713391.148692]  [<ffffffffc02da964>] ? ext4_map_blocks+0x164/0x5d0 [ext4]
[20713391.156273]  [<ffffffffc02db860>] ? ext4_getblk+0x50/0x190 [ext4]
[20713391.163368]  [<ffffffffc02db9bf>] ? ext4_bread+0x1f/0xb0 [ext4]
[20713391.170269]  [<ffffffffc02e54e8>] ? ext4_append+0x48/0xd0 [ext4]
[20713391.177268]  [<ffffffffc02ea956>] ? ext4_mkdir+0x276/0x470 [ext4]
[20713391.184358]  [<ffffffffb9c18e0c>] ? vfs_mkdir+0x10c/0x1a0
[20713391.190673]  [<ffffffffb9c1d7fe>] ? SyS_mkdir+0xce/0x120
[20713391.196890]  [<ffffffffb9a03b7d>] ? do_syscall_64+0x8d/0x100
[20713391.203495]  [<ffffffffba01a88e>] ? entry_SYSCALL_64_after_swapgs+0x58/0xc6
[20713391.211554] Code: 01 00 48 03 04 cd e0 f3 46 ba 48 89 10 8b 42 08 85 c0 75 09 f3 90 8b 42 08 85 c0 74 f7 4c 8b 02 4d 85 c0 74 08 41 0f 18 08 eb 02 <f3> 90 8b 0f 66 85 c9 75 f7 89 c8 66 31 c0 39 c6 74 50 4d 85 c0
[20713394.176861] NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [kworker/u50:8:20807]
[20713394.186096] Modules linked in: cpuid nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter xt_NFLOG xt_limit xt_tcpudp xt_pkttype nf_conntrack_ipv6 nf_defrag_ipv6 binfmt_misc xt_conntrack ip6table_filter ip6_tables nfnetlink_log nfnetlink intel_rapl sb_edac edac_core x86_pkg_temp_thermal intel_powerclamp coretemp kvm mgag200 irqbypass ttm drm_kms_helper drm iTCO_wdt crct10dif_pclmul crc32_pclmul i2c_algo_bit sg iTCO_vendor_support evdev mxm_wmi ghash_clmulni_intel mei_me dcdbas lpc_ich pcspkr mei shpchp mfd_core wmi ipmi_si button ipmi_devintf ipmi_msghandler nf_conntrack ip_tables x_tables autofs4 ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache dm_mod sd_mod crc32c_intel aesni_intel aes_x86_64 glue_helper ahci lrw gf128mul ehci_pci ablk_helper libahci ehci_hcd tg3 cryptd ptp libata megaraid_sas usbcore pps_core usb_common libphy scsi_mod
[20713394.269901] CPU: 9 PID: 20807 Comm: kworker/u50:8 Tainted: G      D      L  4.9.0-9-amd64 #1 Debian 4.9.168-1+deb9u2
[20713394.281934] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.3.4 11/08/2016
[20713394.290773] Workqueue: writeback wb_workfn (flush-8:32)
[20713394.296913] task: ffff9f84363ab140 task.stack: ffffb3284c894000
[20713394.303807] RIP: 0010:[<ffffffffb9ac5b2f>]  [<ffffffffb9ac5b2f>] native_queued_spin_lock_slowpath+0x5f/0x1a0
[20713394.315077] RSP: 0018:ffffb3284c897bc8  EFLAGS: 00000202
[20713394.321293] RAX: 0000000000000101 RBX: ffff9f80d6fbae28 RCX: ffff9f7b22ad7a88
[20713394.329545] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9f7b22ad7b10
[20713394.337796] RBP: ffff9f88db78a800 R08: ffffb3284c897c78 R09: 0000000000ffffff
[20713394.346049] R10: 00000000f0f0f0f0 R11: 00000000fa83b2da R12: ffff9f7b22ad7b78
[20713394.354301] R13: 0000000000000000 R14: ffffb3284c897db0 R15: ffff9f80d6fbae00
[20713394.362555] FS:  0000000000000000(0000) GS:ffff9f88df300000(0000) knlGS:0000000000000000
[20713394.371873] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[20713394.378575] CR2: 00007f849435fa58 CR3: 00000005ddc08000 CR4: 0000000000160670
[20713394.386827] Stack:
[20713394.389359]  ffffffffba01a79d ffffffffb9c3b94e ffff9f80d6fbae58 ffff9f84363ab140
[20713394.397941]  ffff9f80d6fbae38 00000002347bda99 ffff9f7b22ad7b10 ffff9f7b22ad7a88
[20713394.406524]  ffff9f80d6f5e9c0 0000000000000000 0000000000000000 0000000000000000
[20713394.415107] Call Trace:
[20713394.418126]  [<ffffffffba01a79d>] ? _raw_spin_lock+0x1d/0x20
[20713394.424730]  [<ffffffffb9c3b94e>] ? writeback_sb_inodes+0x13e/0x4f0
[20713394.432015]  [<ffffffffb9c3bd87>] ? __writeback_inodes_wb+0x87/0xb0
[20713394.439298]  [<ffffffffb9c3c0fe>] ? wb_writeback+0x27e/0x310
[20713394.445905]  [<ffffffffb9c2798c>] ? get_nr_inodes+0x3c/0x60
[20713394.452413]  [<ffffffffb9c3ca64>] ? wb_workfn+0x2b4/0x380
[20713394.458727]  [<ffffffffb9a9460a>] ? process_one_work+0x18a/0x430
[20713394.465721]  [<ffffffffb9a948fd>] ? worker_thread+0x4d/0x490
[20713394.472326]  [<ffffffffb9a948b0>] ? process_one_work+0x430/0x430
[20713394.479318]  [<ffffffffb9a9a969>] ? kthread+0xd9/0xf0
[20713394.485244]  [<ffffffffba01a9a4>] ? __switch_to_asm+0x34/0x70
[20713394.491946]  [<ffffffffb9a9a890>] ? kthread_park+0x60/0x60
[20713394.498358]  [<ffffffffba01aa37>] ? ret_from_fork+0x57/0x70
[20713394.504864] Code: 75 42 f0 0f ba 2f 08 b8 00 01 00 00 0f 42 f0 8b 07 30 e4 09 c6 f7 c6 00 ff ff ff 75 1b 85 f6 74 0e 8b 07 84 c0 74 08 f3 90 8b 07 <84> c0 75 f8 b8 01 00 00 00 66 89 07 c3 81 e6 00 ff 00 00 75 04
[20713401.501078] INFO: rcu_sched self-detected stall on CPU[20713401.505104] INFO: rcu_sched detected stalls on CPUs/tasks:
[20713401.505112] 	9-...: (2583124 ticks this GP) idle=013/140000000000001/0 softirq=1099266404/1099266404 fqs=1269620
[20713401.505113]
[20713401.505116] (detected by 0, t=2620581 jiffies, g=1081841206, c=1081841205, q=18274458)
[20713401.505117] Task dump for CPU 9:
[20713401.505119] kworker/u50:8   R
[20713401.505120]   running task        0 20807      2 0x00000008
[20713401.505130] Workqueue: writeback wb_workfn
[20713401.505131]  (flush-8:32) 0000000000000000
[20713401.505134]  0000000000000000 0000000000000000 fcb9f935fd90189d ffff9f80d6fbaf88
[20713401.505138]  ffff9f843a75f000 ffff9f88ded2e800 ffff9f88db239b00 0000000000000000
[20713401.505141]  ffff9f80d6fbaf90 ffffffffb9a9460a 00000000363ab140Call Trace:
[20713401.505151]  [<ffffffffb9a9460a>] ? process_one_work+0x18a/0x430
[20713401.505155]  [<ffffffffb9a948fd>] ? worker_thread+0x4d/0x490
[20713401.505158]  [<ffffffffb9a948b0>] ? process_one_work+0x430/0x430
[20713401.505162]  [<ffffffffb9a9a969>] ? kthread+0xd9/0xf0
[20713401.505166]  [<ffffffffba01a9a4>] ? __switch_to_asm+0x34/0x70
[20713401.505170]  [<ffffffffb9a9a890>] ? kthread_park+0x60/0x60
[20713401.505173]  [<ffffffffba01aa37>] ? ret_from_fork+0x57/0x70
[20713401.631216] 	9-...: (2583124 ticks this GP) idle=013/140000000000001/0 softirq=1099266404/1099266404 fqs=1269636
[20713401.642958] 	 (t=2620615 jiffies g=1081841206 c=1081841205 q=18274458)
[20713401.650542] Task dump for CPU 9:
[20713401.654430] kworker/u50:8   R  running task        0 20807      2 0x00000008
[20713401.662609] Workqueue: writeback wb_workfn (flush-8:32)
[20713401.668749]  ffffffffba7198c0 ffffffffb9aa822b 0000000000000009 ffffffffba7198c0
[20713401.677332]  ffffffffb9b81a3e ffff9f88df3196c0 ffffffffba64fd40 0000000000000000
[20713401.685912]  ffffffffba7198c0 00000000ffffffff ffffffffb9ae3cba 0000000000000001
[20713401.694493] Call Trace:
[20713401.697509]  <IRQ> [20713401.699948]  [<ffffffffb9aa822b>] ? sched_show_task+0xcb/0x130
[20713401.706754]  [<ffffffffb9b81a3e>] ? rcu_dump_cpu_stacks+0x92/0xb2
[20713401.713847]  [<ffffffffb9ae3cba>] ? rcu_check_callbacks+0x75a/0x8b0
[20713401.721132]  [<ffffffffb9afa260>] ? tick_sched_do_timer+0x30/0x30
[20713401.728221]  [<ffffffffb9aea8a8>] ? update_process_times+0x28/0x50
[20713401.735408]  [<ffffffffb9af9c60>] ? tick_sched_handle.isra.12+0x20/0x50
[20713401.743079]  [<ffffffffb9afa298>] ? tick_sched_timer+0x38/0x70
[20713401.749878]  [<ffffffffb9aeb37e>] ? __hrtimer_run_queues+0xde/0x250
[20713401.757162]  [<ffffffffb9aeba5c>] ? hrtimer_interrupt+0x9c/0x1a0
[20713401.764155]  [<ffffffffba01de17>] ? smp_apic_timer_interrupt+0x47/0x60
[20713401.771731]  [<ffffffffba01c6a6>] ? apic_timer_interrupt+0x96/0xa0
[20713401.778916]  <EOI> [20713401.781352]  [<ffffffffb9ac5b2b>] ? native_queued_spin_lock_slowpath+0x5b/0x1a0
[20713401.789806]  [<ffffffffba01a79d>] ? _raw_spin_lock+0x1d/0x20
[20713401.796411]  [<ffffffffb9c3b94e>] ? writeback_sb_inodes+0x13e/0x4f0
[20713401.803694]  [<ffffffffb9c3bd87>] ? __writeback_inodes_wb+0x87/0xb0
[20713401.810977]  [<ffffffffb9c3c0fe>] ? wb_writeback+0x27e/0x310
[20713401.817583]  [<ffffffffb9c2798c>] ? get_nr_inodes+0x3c/0x60
[20713401.824091]  [<ffffffffb9c3ca64>] ? wb_workfn+0x2b4/0x380
[20713401.830405]  [<ffffffffb9a9460a>] ? process_one_work+0x18a/0x430
[20713401.837396]  [<ffffffffb9a948fd>] ? worker_thread+0x4d/0x490
[20713401.843999]  [<ffffffffb9a948b0>] ? process_one_work+0x430/0x430
[20713401.850993]  [<ffffffffb9a9a969>] ? kthread+0xd9/0xf0
[20713401.856920]  [<ffffffffba01a9a4>] ? __switch_to_asm+0x34/0x70
[20713401.863623]  [<ffffffffb9a9a890>] ? kthread_park+0x60/0x60
[20713401.870032]  [<ffffffffba01aa37>] ? ret_from_fork+0x57/0x70

Event Timeline

jijiki created this task.Sun, Feb 2, 8:46 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSun, Feb 2, 8:46 AM
jijiki triaged this task as Medium priority.Sun, Feb 2, 8:47 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-02T08:48:28Z] <effie> reboot host analytics1061 - T244081

elukey closed this task as Resolved.Tue, Feb 4, 9:36 AM
elukey claimed this task.

Didn't re-occur, will re-open if needed.