Paste P6441

mlr training container kill / nodemanager log
ActivePublic

Authored by EBernhardson on Dec 7 2017, 5:05 AM.
1​2017-12-07 04:22:56,995 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e54_1512469367986_4908_01_000007 by user ebernhardson
2​2017-12-07 04:22:56,996 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=ebernhardson IP=10.64.21.105 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1512469367986_4908 CONTAINERID=container_e54_1512469367986_4908_01_000007
3​2017-12-07 04:22:57,003 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_e54_1512469367986_4908_01_000007 to application application_1512469367986_4908
4​2017-12-07 04:22:57,004 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from NEW to LOCALIZING
5​2017-12-07 04:22:57,004 INFO org.apache.spark.network.yarn.YarnShuffleService: Initializing container container_e54_1512469367986_4908_01_000007
6​2017-12-07 04:22:57,004 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e54_1512469367986_4908_01_000007
7​2017-12-07 04:22:57,006 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /var/lib/hadoop/data/b/yarn/local/nmPrivate/container_e54_1512469367986_4908_01_000007.tokens. Credentials list:
8​2017-12-07 04:22:57,158 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /var/lib/hadoop/data/b/yarn/local/nmPrivate/container_e54_1512469367986_4908_01_000007.tokens to /var/lib/hadoop/data/l/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007.tokens
9​2017-12-07 04:23:18,879 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from LOCALIZING to LOCALIZED
10​2017-12-07 04:23:18,907 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from LOCALIZED to RUNNING
11​2017-12-07 04:23:18,912 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/default_container_executor.sh]
12​2017-12-07 04:23:19,896 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e54_1512469367986_4908_01_000007
13​2017-12-07 04:23:20,036 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 140.2 MB of 10 GB physical memory used; 4.8 GB of 21 GB virtual memory used
14​2017-12-07 04:23:23,097 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 427.0 MB of 10 GB physical memory used; 5.0 GB of 21 GB virtual memory used
15​2017-12-07 04:23:26,155 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 427.2 MB of 10 GB physical memory used; 5.0 GB of 21 GB virtual memory used
16​2017-12-07 04:23:29,210 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 427.2 MB of 10 GB physical memory used; 5.0 GB of 21 GB virtual memory used
17​2017-12-07 04:23:32,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 585.9 MB of 10 GB physical memory used; 5.0 GB of 21 GB virtual memory used
18​2017-12-07 04:23:35,336 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 782.3 MB of 10 GB physical memory used; 5.0 GB of 21 GB virtual memory used
19​2017-12-07 04:23:38,402 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 879.0 MB of 10 GB physical memory used; 5.1 GB of 21 GB virtual memory used
20​2017-12-07 04:23:41,468 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 879.3 MB of 10 GB physical memory used; 5.1 GB of 21 GB virtual memory used
21​2017-12-07 04:23:44,532 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 879.3 MB of 10 GB physical memory used; 5.1 GB of 21 GB virtual memory used
22​2017-12-07 04:23:47,596 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 1.1 GB of 10 GB physical memory used; 6.9 GB of 21 GB virtual memory used
23​2017-12-07 04:23:50,655 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 1.7 GB of 10 GB physical memory used; 6.9 GB of 21 GB virtual memory used
24​2017-12-07 04:23:53,707 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 1.8 GB of 10 GB physical memory used; 6.9 GB of 21 GB virtual memory used
25​2017-12-07 04:23:56,761 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 2.3 GB of 10 GB physical memory used; 7.0 GB of 21 GB virtual memory used
26​2017-12-07 04:23:59,817 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 2.3 GB of 10 GB physical memory used; 7.0 GB of 21 GB virtual memory used
27​2017-12-07 04:24:02,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 2.5 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
28​2017-12-07 04:24:05,919 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 3.2 GB of 10 GB physical memory used; 7.2 GB of 21 GB virtual memory used
29​2017-12-07 04:24:08,975 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 3.8 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
30​2017-12-07 04:24:12,028 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 3.8 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
31​2017-12-07 04:24:15,081 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.2 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
32​2017-12-07 04:24:18,133 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.4 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
33​2017-12-07 04:24:21,186 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.4 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
34​2017-12-07 04:24:24,230 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
35​2017-12-07 04:24:27,277 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
36​2017-12-07 04:24:30,332 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
37​2017-12-07 04:24:33,384 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
38​2017-12-07 04:24:36,440 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
39​2017-12-07 04:24:39,496 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
40​2017-12-07 04:24:42,550 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
41​2017-12-07 04:24:45,601 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
42​2017-12-07 04:24:48,652 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
43​2017-12-07 04:24:51,705 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
44​2017-12-07 04:24:54,749 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
45​2017-12-07 04:24:57,802 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
46​2017-12-07 04:25:00,860 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
47​2017-12-07 04:25:03,919 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
48​2017-12-07 04:25:06,978 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
49​2017-12-07 04:25:10,037 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
50​2017-12-07 04:25:13,094 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
51​2017-12-07 04:25:16,146 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
52​2017-12-07 04:25:19,287 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.9 GB of 10 GB physical memory used; 7.4 GB of 21 GB virtual memory used
53​2017-12-07 04:25:22,341 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 5.1 GB of 10 GB physical memory used; 7.8 GB of 21 GB virtual memory used
54​2017-12-07 04:25:25,393 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 5.5 GB of 10 GB physical memory used; 8.7 GB of 21 GB virtual memory used
55​2017-12-07 04:25:28,445 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 6.0 GB of 10 GB physical memory used; 9.5 GB of 21 GB virtual memory used
56​2017-12-07 04:25:31,514 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 6.4 GB of 10 GB physical memory used; 10.4 GB of 21 GB virtual memory used
57​2017-12-07 04:25:34,582 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 7.3 GB of 10 GB physical memory used; 11.0 GB of 21 GB virtual memory used
58​2017-12-07 04:25:37,650 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 7.9 GB of 10 GB physical memory used; 11.7 GB of 21 GB virtual memory used
59​2017-12-07 04:25:40,719 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 8.2 GB of 10 GB physical memory used; 12.4 GB of 21 GB virtual memory used
60​2017-12-07 04:25:43,791 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 8.6 GB of 10 GB physical memory used; 12.7 GB of 21 GB virtual memory used
61​2017-12-07 04:25:46,865 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 8.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
62​2017-12-07 04:25:49,943 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.2 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
63​2017-12-07 04:25:53,013 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.5 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
64​2017-12-07 04:25:56,085 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
65​2017-12-07 04:25:59,161 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
66​2017-12-07 04:26:02,231 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
67​2017-12-07 04:26:05,309 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
68​2017-12-07 04:26:08,380 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
69​2017-12-07 04:26:11,451 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
70​2017-12-07 04:26:14,523 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
71​2017-12-07 04:26:17,596 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
72​2017-12-07 04:26:20,668 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
73​2017-12-07 04:26:23,731 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
74​2017-12-07 04:26:26,793 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
75​2017-12-07 04:26:29,861 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
76​2017-12-07 04:26:32,913 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
77​2017-12-07 04:26:35,963 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
78​2017-12-07 04:26:39,014 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
79​2017-12-07 04:26:42,066 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
80​2017-12-07 04:26:45,123 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
81​2017-12-07 04:26:48,178 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
82​2017-12-07 04:26:51,229 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
83​2017-12-07 04:26:54,286 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
84​2017-12-07 04:26:57,341 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
85​2017-12-07 04:27:00,396 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
86​2017-12-07 04:27:03,449 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
87​2017-12-07 04:27:06,499 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
88​2017-12-07 04:27:09,551 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
89​2017-12-07 04:27:12,675 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
90​2017-12-07 04:27:15,747 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
91​2017-12-07 04:27:18,813 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
92​2017-12-07 04:27:21,881 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
93​2017-12-07 04:27:24,951 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
94​2017-12-07 04:27:28,018 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
95​2017-12-07 04:27:31,082 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
96​2017-12-07 04:27:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
97​2017-12-07 04:27:37,223 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
98​2017-12-07 04:27:40,290 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
99​2017-12-07 04:27:43,356 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
100​2017-12-07 04:27:46,422 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
101​2017-12-07 04:27:49,490 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
102​2017-12-07 04:27:52,556 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
103​2017-12-07 04:27:55,623 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
104​2017-12-07 04:27:58,692 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
105​2017-12-07 04:28:01,767 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
106​2017-12-07 04:28:04,835 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
107​2017-12-07 04:28:07,883 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
108​2017-12-07 04:28:10,952 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
109​2017-12-07 04:28:13,994 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
110​2017-12-07 04:28:17,044 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
111​2017-12-07 04:28:20,101 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
112​2017-12-07 04:28:23,154 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
113​2017-12-07 04:28:26,205 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.6 GB of 21 GB virtual memory used
114​2017-12-07 04:28:29,257 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 10.2 GB of 10 GB physical memory used; 12.6 GB of 21 GB virtual memory used
115​2017-12-07 04:28:29,257 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Process tree for container: container_e54_1512469367986_4908_01_000007 has processes older than 1 iteration running over the configured limit. Limit=10737418240, current usage = 10931814400
116​2017-12-07 04:28:29,257 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=25864,containerID=container_e54_1512469367986_4908_01_000007] is running beyond physical memory limits. Current usage: 10.2 GB of 10 GB physical memory used; 12.6 GB of 21 GB virtual memory used. Killing container.
117​Dump of the process-tree for container_e54_1512469367986_4908_01_000007 :
118​ |- 25864 25860 25864 25864 (bash) 0 1 13725696 799 /bin/bash -c /usr/lib/jvm/default-java/bin/java -server -Xmx4096m -Djava.io.tmpdir=/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/tmp '-Dspark.driver.port=40931' -Dspark.yarn.app.container.log.dir=/var/lib/hadoop/data/d/yarn/logs/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007 -XX:MaxPermSize=256m -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.64.53.30:40931 --executor-id 6 --hostname analytics1049.eqiad.wmnet --cores 6 --app-id application_1512469367986_4908 --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/__app__.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/ml.dmlc_xgboost4j-spark-0.7-wmf-2-SNAPSHOT.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.wikimedia.search_mjolnir-0.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.spark_spark-streaming-kafka-0-8_2.11-2.1.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/ml.dmlc_xgboost4j-0.7-wmf-2-SNAPSHOT.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.esotericsoftware.kryo_kryo-2.21.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang_scala-compiler-2.11.8.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang_scala-reflect-2.11.8.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/commons-logging_commons-logging-1.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.typesafe.akka_akka-actor_2.11-2.3.11.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.typesafe_config-1.2.1.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.esotericsoftware.reflectasm_reflectasm-1.07.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.esotericsoftware.minlog_minlog-1.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.objenesis_objenesis-1.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.ow2.asm_asm-4.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang.modules_scala-xml_2.11-1.0.4.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang.modules_scala-parser-combinators_2.11-1.0.4.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.kafka_kafka_2.11-0.8.2.1.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.spark_spark-tags_2.11-2.1.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.spark-project.spark_unused-1.0.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.yammer.metrics_metrics-core-2.2.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.101tec_zkclient-0.3.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.kafka_kafka-clients-0.8.2.1.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.slf4j_slf4j-api-1.7.16.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/log4j_log4j-1.2.17.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/net.jpountz.lz4_lz4-1.3.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.xerial.snappy_snappy-java-1.1.2.6.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scalatest_scalatest_2.11-2.2.6.jar 1>/var/lib/hadoop/data/d/yarn/logs/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/stdout 2>/var/lib/hadoop/data/d/yarn/logs/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/stderr
119​ |- 25933 25864 25864 25864 (java) 27138 3396 12533579776 2654552 /usr/lib/jvm/default-java/bin/java -server -Xmx4096m -Djava.io.tmpdir=/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/tmp -Dspark.driver.port=40931 -Dspark.yarn.app.container.log.dir=/var/lib/hadoop/data/d/yarn/logs/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007 -XX:MaxPermSize=256m -XX:OnOutOfMemoryError=kill %p org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.64.53.30:40931 --executor-id 6 --hostname analytics1049.eqiad.wmnet --cores 6 --app-id application_1512469367986_4908 --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/__app__.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/ml.dmlc_xgboost4j-spark-0.7-wmf-2-SNAPSHOT.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.wikimedia.search_mjolnir-0.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.spark_spark-streaming-kafka-0-8_2.11-2.1.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/ml.dmlc_xgboost4j-0.7-wmf-2-SNAPSHOT.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.esotericsoftware.kryo_kryo-2.21.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang_scala-compiler-2.11.8.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang_scala-reflect-2.11.8.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/commons-logging_commons-logging-1.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.typesafe.akka_akka-actor_2.11-2.3.11.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.typesafe_config-1.2.1.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.esotericsoftware.reflectasm_reflectasm-1.07.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.esotericsoftware.minlog_minlog-1.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.objenesis_objenesis-1.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.ow2.asm_asm-4.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang.modules_scala-xml_2.11-1.0.4.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang.modules_scala-parser-combinators_2.11-1.0.4.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.kafka_kafka_2.11-0.8.2.1.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.spark_spark-tags_2.11-2.1.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.spark-project.spark_unused-1.0.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.yammer.metrics_metrics-core-2.2.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.101tec_zkclient-0.3.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.kafka_kafka-clients-0.8.2.1.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.slf4j_slf4j-api-1.7.16.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/log4j_log4j-1.2.17.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/net.jpountz.lz4_lz4-1.3.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.xerial.snappy_snappy-java-1.1.2.6.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scalatest_scalatest_2.11-2.2.6.jar
120​2017-12-07 04:28:29,258 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from RUNNING to KILLING
121​2017-12-07 04:28:29,258 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e54_1512469367986_4908_01_000007
122​2017-12-07 04:28:29,268 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e54_1512469367986_4908_01_000007 is : 143
123​2017-12-07 04:28:29,293 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL
124​2017-12-07 04:28:29,293 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/b/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
125​2017-12-07 04:28:29,293 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/c/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
126​2017-12-07 04:28:29,293 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/d/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
127​2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/e/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
128​2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
129​2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/g/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
130​2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/h/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
131​2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/i/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
132​2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/j/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
133​2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/k/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
134​2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/l/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
135​2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/m/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
136​2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=ebernhardson OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1512469367986_4908 CONTAINERID=container_e54_1512469367986_4908_01_000007
137​2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE
138​2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_e54_1512469367986_4908_01_000007 from application application_1512469367986_4908
139​2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e54_1512469367986_4908_01_000007 for log-aggregation
140​2017-12-07 04:28:29,295 INFO org.apache.spark.network.yarn.YarnShuffleService: Stopping container container_e54_1512469367986_4908_01_000007
141​2017-12-07 04:28:30,301 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e54_1512469367986_4908_01_000007]
142​2017-12-07 04:28:32,274 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e54_1512469367986_4908_01_000007
143​2017-12-07 04:48:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e54_1512469367986_4908_01_000007. Current good log dirs are /var/lib/hadoop/data/b/yarn/logs,/var/lib/hadoop/data/c/yarn/logs,/var/lib/hadoop/data/d/yarn/logs,/var/lib/hadoop/data/e/yarn/logs,/var/lib/hadoop/data/f/yarn/logs,/var/lib/hadoop/data/g/yarn/logs,/var/lib/hadoop/data/h/yarn/logs,/var/lib/hadoop/data/i/yarn/logs,/var/lib/hadoop/data/j/yarn/logs,/var/lib/hadoop/data/k/yarn/logs,/var/lib/hadoop/data/l/yarn/logs,/var/lib/hadoop/data/m/yarn/logs
144​2017-12-07 04:48:37,960 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/lib/hadoop/data/d/yarn/logs/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/stdout
145​2017-12-07 04:48:37,961 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/lib/hadoop/data/d/yarn/logs/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/stderr