Paste P6441

mlr training container kill / nodemanager log
ActivePublic

Authored by EBernhardson on Dec 7 2017, 5:05 AM.
2017-12-07 04:22:56,995 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e54_1512469367986_4908_01_000007 by user ebernhardson
2017-12-07 04:22:56,996 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=ebernhardson IP=10.64.21.105 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1512469367986_4908 CONTAINERID=container_e54_1512469367986_4908_01_000007
2017-12-07 04:22:57,003 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_e54_1512469367986_4908_01_000007 to application application_1512469367986_4908
2017-12-07 04:22:57,004 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from NEW to LOCALIZING
2017-12-07 04:22:57,004 INFO org.apache.spark.network.yarn.YarnShuffleService: Initializing container container_e54_1512469367986_4908_01_000007
2017-12-07 04:22:57,004 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e54_1512469367986_4908_01_000007
2017-12-07 04:22:57,006 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /var/lib/hadoop/data/b/yarn/local/nmPrivate/container_e54_1512469367986_4908_01_000007.tokens. Credentials list:
2017-12-07 04:22:57,158 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /var/lib/hadoop/data/b/yarn/local/nmPrivate/container_e54_1512469367986_4908_01_000007.tokens to /var/lib/hadoop/data/l/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007.tokens
2017-12-07 04:23:18,879 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from LOCALIZING to LOCALIZED
2017-12-07 04:23:18,907 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from LOCALIZED to RUNNING
2017-12-07 04:23:18,912 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/default_container_executor.sh]
2017-12-07 04:23:19,896 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e54_1512469367986_4908_01_000007
2017-12-07 04:23:20,036 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 140.2 MB of 10 GB physical memory used; 4.8 GB of 21 GB virtual memory used
2017-12-07 04:23:23,097 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 427.0 MB of 10 GB physical memory used; 5.0 GB of 21 GB virtual memory used
2017-12-07 04:23:26,155 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 427.2 MB of 10 GB physical memory used; 5.0 GB of 21 GB virtual memory used
2017-12-07 04:23:29,210 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 427.2 MB of 10 GB physical memory used; 5.0 GB of 21 GB virtual memory used
2017-12-07 04:23:32,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 585.9 MB of 10 GB physical memory used; 5.0 GB of 21 GB virtual memory used
2017-12-07 04:23:35,336 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 782.3 MB of 10 GB physical memory used; 5.0 GB of 21 GB virtual memory used
2017-12-07 04:23:38,402 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 879.0 MB of 10 GB physical memory used; 5.1 GB of 21 GB virtual memory used
2017-12-07 04:23:41,468 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 879.3 MB of 10 GB physical memory used; 5.1 GB of 21 GB virtual memory used
2017-12-07 04:23:44,532 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 879.3 MB of 10 GB physical memory used; 5.1 GB of 21 GB virtual memory used
2017-12-07 04:23:47,596 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 1.1 GB of 10 GB physical memory used; 6.9 GB of 21 GB virtual memory used
2017-12-07 04:23:50,655 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 1.7 GB of 10 GB physical memory used; 6.9 GB of 21 GB virtual memory used
2017-12-07 04:23:53,707 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 1.8 GB of 10 GB physical memory used; 6.9 GB of 21 GB virtual memory used
2017-12-07 04:23:56,761 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 2.3 GB of 10 GB physical memory used; 7.0 GB of 21 GB virtual memory used
2017-12-07 04:23:59,817 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 2.3 GB of 10 GB physical memory used; 7.0 GB of 21 GB virtual memory used
2017-12-07 04:24:02,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 2.5 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:05,919 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 3.2 GB of 10 GB physical memory used; 7.2 GB of 21 GB virtual memory used
2017-12-07 04:24:08,975 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 3.8 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:12,028 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 3.8 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:15,081 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.2 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:18,133 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.4 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:21,186 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.4 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:24,230 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:27,277 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:30,332 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:33,384 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:36,440 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:39,496 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:42,550 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:45,601 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:48,652 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:51,705 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:54,749 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:57,802 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:25:00,860 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:25:03,919 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:25:06,978 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:25:10,037 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:25:13,094 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:25:16,146 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:25:19,287 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.9 GB of 10 GB physical memory used; 7.4 GB of 21 GB virtual memory used
2017-12-07 04:25:22,341 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 5.1 GB of 10 GB physical memory used; 7.8 GB of 21 GB virtual memory used
2017-12-07 04:25:25,393 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 5.5 GB of 10 GB physical memory used; 8.7 GB of 21 GB virtual memory used
2017-12-07 04:25:28,445 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 6.0 GB of 10 GB physical memory used; 9.5 GB of 21 GB virtual memory used
2017-12-07 04:25:31,514 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 6.4 GB of 10 GB physical memory used; 10.4 GB of 21 GB virtual memory used
2017-12-07 04:25:34,582 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 7.3 GB of 10 GB physical memory used; 11.0 GB of 21 GB virtual memory used
2017-12-07 04:25:37,650 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 7.9 GB of 10 GB physical memory used; 11.7 GB of 21 GB virtual memory used
2017-12-07 04:25:40,719 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 8.2 GB of 10 GB physical memory used; 12.4 GB of 21 GB virtual memory used
2017-12-07 04:25:43,791 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 8.6 GB of 10 GB physical memory used; 12.7 GB of 21 GB virtual memory used
2017-12-07 04:25:46,865 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 8.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:25:49,943 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.2 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:25:53,013 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.5 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:25:56,085 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:25:59,161 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:02,231 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:05,309 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:08,380 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:11,451 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:14,523 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:17,596 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:20,668 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:23,731 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:26,793 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:29,861 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:32,913 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:35,963 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:39,014 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:42,066 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:45,123 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:48,178 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:51,229 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:54,286 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:57,341 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:00,396 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:03,449 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:06,499 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:09,551 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:12,675 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:15,747 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:18,813 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:21,881 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:24,951 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:28,018 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:31,082 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:37,223 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:40,290 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:43,356 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:46,422 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:49,490 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:52,556 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:55,623 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:58,692 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:01,767 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:04,835 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:07,883 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:10,952 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:13,994 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:17,044 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:20,101 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:23,154 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:26,205 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.6 GB of 21 GB virtual memory used
2017-12-07 04:28:29,257 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 10.2 GB of 10 GB physical memory used; 12.6 GB of 21 GB virtual memory used
2017-12-07 04:28:29,257 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Process tree for container: container_e54_1512469367986_4908_01_000007 has processes older than 1 iteration running over the configured limit. Limit=10737418240, current usage = 10931814400
2017-12-07 04:28:29,257 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=25864,containerID=container_e54_1512469367986_4908_01_000007] is running beyond physical memory limits. Current usage: 10.2 GB of 10 GB physical memory used; 12.6 GB of 21 GB virtual memory used. Killing container.
Dump of the process-tree for container_e54_1512469367986_4908_01_000007 :
|- 25864 25860 25864 25864 (bash) 0 1 13725696 799 /bin/bash -c /usr/lib/jvm/default-java/bin/java -server -Xmx4096m -Djava.io.tmpdir=/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/tmp '-Dspark.driver.port=40931' -Dspark.yarn.app.container.log.dir=/var/lib/hadoop/data/d/yarn/logs/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007 -XX:MaxPermSize=256m -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.64.53.30:40931 --executor-id 6 --hostname analytics1049.eqiad.wmnet --cores 6 --app-id application_1512469367986_4908 --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/__app__.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/ml.dmlc_xgboost4j-spark-0.7-wmf-2-SNAPSHOT.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.wikimedia.search_mjolnir-0.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.spark_spark-streaming-kafka-0-8_2.11-2.1.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/ml.dmlc_xgboost4j-0.7-wmf-2-SNAPSHOT.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.esotericsoftware.kryo_kryo-2.21.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang_scala-compiler-2.11.8.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang_scala-reflect-2.11.8.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/commons-logging_commons-logging-1.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.typesafe.akka_akka-actor_2.11-2.3.11.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.typesafe_config-1.2.1.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.esotericsoftware.reflectasm_reflectasm-1.07.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.esotericsoftware.minlog_minlog-1.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.objenesis_objenesis-1.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.ow2.asm_asm-4.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang.modules_scala-xml_2.11-1.0.4.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang.modules_scala-parser-combinators_2.11-1.0.4.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.kafka_kafka_2.11-0.8.2.1.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.spark_spark-tags_2.11-2.1.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.spark-project.spark_unused-1.0.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.yammer.metrics_metrics-core-2.2.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.101tec_zkclient-0.3.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.kafka_kafka-clients-0.8.2.1.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.slf4j_slf4j-api-1.7.16.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/log4j_log4j-1.2.17.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/net.jpountz.lz4_lz4-1.3.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.xerial.snappy_snappy-java-1.1.2.6.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scalatest_scalatest_2.11-2.2.6.jar 1>/var/lib/hadoop/data/d/yarn/logs/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/stdout 2>/var/lib/hadoop/data/d/yarn/logs/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/stderr
|- 25933 25864 25864 25864 (java) 27138 3396 12533579776 2654552 /usr/lib/jvm/default-java/bin/java -server -Xmx4096m -Djava.io.tmpdir=/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/tmp -Dspark.driver.port=40931 -Dspark.yarn.app.container.log.dir=/var/lib/hadoop/data/d/yarn/logs/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007 -XX:MaxPermSize=256m -XX:OnOutOfMemoryError=kill %p org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.64.53.30:40931 --executor-id 6 --hostname analytics1049.eqiad.wmnet --cores 6 --app-id application_1512469367986_4908 --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/__app__.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/ml.dmlc_xgboost4j-spark-0.7-wmf-2-SNAPSHOT.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.wikimedia.search_mjolnir-0.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.spark_spark-streaming-kafka-0-8_2.11-2.1.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/ml.dmlc_xgboost4j-0.7-wmf-2-SNAPSHOT.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.esotericsoftware.kryo_kryo-2.21.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang_scala-compiler-2.11.8.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang_scala-reflect-2.11.8.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/commons-logging_commons-logging-1.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.typesafe.akka_akka-actor_2.11-2.3.11.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.typesafe_config-1.2.1.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.esotericsoftware.reflectasm_reflectasm-1.07.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.esotericsoftware.minlog_minlog-1.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.objenesis_objenesis-1.2.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.ow2.asm_asm-4.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang.modules_scala-xml_2.11-1.0.4.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scala-lang.modules_scala-parser-combinators_2.11-1.0.4.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.kafka_kafka_2.11-0.8.2.1.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.spark_spark-tags_2.11-2.1.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.spark-project.spark_unused-1.0.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.yammer.metrics_metrics-core-2.2.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/com.101tec_zkclient-0.3.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.apache.kafka_kafka-clients-0.8.2.1.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.slf4j_slf4j-api-1.7.16.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/log4j_log4j-1.2.17.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/net.jpountz.lz4_lz4-1.3.0.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.xerial.snappy_snappy-java-1.1.2.6.jar --user-class-path file:/var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/org.scalatest_scalatest_2.11-2.2.6.jar
2017-12-07 04:28:29,258 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from RUNNING to KILLING
2017-12-07 04:28:29,258 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,268 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e54_1512469367986_4908_01_000007 is : 143
2017-12-07 04:28:29,293 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL
2017-12-07 04:28:29,293 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/b/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,293 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/c/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,293 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/d/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/e/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/g/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/h/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/i/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/j/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/k/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/l/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/m/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=ebernhardson OPERATION=Container Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS APPID=application_1512469367986_4908 CONTAINERID=container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE
2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_e54_1512469367986_4908_01_000007 from application application_1512469367986_4908
2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e54_1512469367986_4908_01_000007 for log-aggregation
2017-12-07 04:28:29,295 INFO org.apache.spark.network.yarn.YarnShuffleService: Stopping container container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:30,301 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e54_1512469367986_4908_01_000007]
2017-12-07 04:28:32,274 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e54_1512469367986_4908_01_000007
2017-12-07 04:48:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e54_1512469367986_4908_01_000007. Current good log dirs are /var/lib/hadoop/data/b/yarn/logs,/var/lib/hadoop/data/c/yarn/logs,/var/lib/hadoop/data/d/yarn/logs,/var/lib/hadoop/data/e/yarn/logs,/var/lib/hadoop/data/f/yarn/logs,/var/lib/hadoop/data/g/yarn/logs,/var/lib/hadoop/data/h/yarn/logs,/var/lib/hadoop/data/i/yarn/logs,/var/lib/hadoop/data/j/yarn/logs,/var/lib/hadoop/data/k/yarn/logs,/var/lib/hadoop/data/l/yarn/logs,/var/lib/hadoop/data/m/yarn/logs
2017-12-07 04:48:37,960 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/lib/hadoop/data/d/yarn/logs/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/stdout
2017-12-07 04:48:37,961 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/lib/hadoop/data/d/yarn/logs/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/stderr