2017-12-07 04:22:56,995 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_e54_1512469367986_4908_01_000007 by user ebernhardson
2017-12-07 04:22:57,003 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_e54_1512469367986_4908_01_000007 to application application_1512469367986_4908
2017-12-07 04:22:57,004 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from NEW to LOCALIZING
2017-12-07 04:22:57,004 INFO org.apache.spark.network.yarn.YarnShuffleService: Initializing container container_e54_1512469367986_4908_01_000007
2017-12-07 04:22:57,004 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Created localizer for container_e54_1512469367986_4908_01_000007
2017-12-07 04:22:57,006 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Writing credentials to the nmPrivate file /var/lib/hadoop/data/b/yarn/local/nmPrivate/container_e54_1512469367986_4908_01_000007.tokens. Credentials list:
2017-12-07 04:22:57,158 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Copying from /var/lib/hadoop/data/b/yarn/local/nmPrivate/container_e54_1512469367986_4908_01_000007.tokens to /var/lib/hadoop/data/l/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007.tokens
2017-12-07 04:23:18,879 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from LOCALIZING to LOCALIZED
2017-12-07 04:23:18,907 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from LOCALIZED to RUNNING
2017-12-07 04:23:18,912 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, /var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/default_container_executor.sh]
2017-12-07 04:23:19,896 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_e54_1512469367986_4908_01_000007
2017-12-07 04:23:20,036 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 140.2 MB of 10 GB physical memory used; 4.8 GB of 21 GB virtual memory used
2017-12-07 04:23:23,097 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 427.0 MB of 10 GB physical memory used; 5.0 GB of 21 GB virtual memory used
2017-12-07 04:23:26,155 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 427.2 MB of 10 GB physical memory used; 5.0 GB of 21 GB virtual memory used
2017-12-07 04:23:29,210 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 427.2 MB of 10 GB physical memory used; 5.0 GB of 21 GB virtual memory used
2017-12-07 04:23:32,268 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 585.9 MB of 10 GB physical memory used; 5.0 GB of 21 GB virtual memory used
2017-12-07 04:23:35,336 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 782.3 MB of 10 GB physical memory used; 5.0 GB of 21 GB virtual memory used
2017-12-07 04:23:38,402 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 879.0 MB of 10 GB physical memory used; 5.1 GB of 21 GB virtual memory used
2017-12-07 04:23:41,468 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 879.3 MB of 10 GB physical memory used; 5.1 GB of 21 GB virtual memory used
2017-12-07 04:23:44,532 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 879.3 MB of 10 GB physical memory used; 5.1 GB of 21 GB virtual memory used
2017-12-07 04:23:47,596 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 1.1 GB of 10 GB physical memory used; 6.9 GB of 21 GB virtual memory used
2017-12-07 04:23:50,655 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 1.7 GB of 10 GB physical memory used; 6.9 GB of 21 GB virtual memory used
2017-12-07 04:23:53,707 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 1.8 GB of 10 GB physical memory used; 6.9 GB of 21 GB virtual memory used
2017-12-07 04:23:56,761 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 2.3 GB of 10 GB physical memory used; 7.0 GB of 21 GB virtual memory used
2017-12-07 04:23:59,817 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 2.3 GB of 10 GB physical memory used; 7.0 GB of 21 GB virtual memory used
2017-12-07 04:24:02,867 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 2.5 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:05,919 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 3.2 GB of 10 GB physical memory used; 7.2 GB of 21 GB virtual memory used
2017-12-07 04:24:08,975 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 3.8 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:12,028 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 3.8 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:15,081 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.2 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:18,133 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.4 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:21,186 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.4 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:24,230 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:27,277 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:30,332 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:33,384 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:36,440 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:39,496 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:42,550 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:45,601 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:48,652 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:51,705 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:54,749 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:24:57,802 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:25:00,860 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:25:03,919 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:25:06,978 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:25:10,037 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:25:13,094 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:25:16,146 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.6 GB of 10 GB physical memory used; 7.1 GB of 21 GB virtual memory used
2017-12-07 04:25:19,287 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 4.9 GB of 10 GB physical memory used; 7.4 GB of 21 GB virtual memory used
2017-12-07 04:25:22,341 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 5.1 GB of 10 GB physical memory used; 7.8 GB of 21 GB virtual memory used
2017-12-07 04:25:25,393 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 5.5 GB of 10 GB physical memory used; 8.7 GB of 21 GB virtual memory used
2017-12-07 04:25:28,445 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 6.0 GB of 10 GB physical memory used; 9.5 GB of 21 GB virtual memory used
2017-12-07 04:25:31,514 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 6.4 GB of 10 GB physical memory used; 10.4 GB of 21 GB virtual memory used
2017-12-07 04:25:34,582 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 7.3 GB of 10 GB physical memory used; 11.0 GB of 21 GB virtual memory used
2017-12-07 04:25:37,650 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 7.9 GB of 10 GB physical memory used; 11.7 GB of 21 GB virtual memory used
2017-12-07 04:25:40,719 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 8.2 GB of 10 GB physical memory used; 12.4 GB of 21 GB virtual memory used
2017-12-07 04:25:43,791 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 8.6 GB of 10 GB physical memory used; 12.7 GB of 21 GB virtual memory used
2017-12-07 04:25:46,865 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 8.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:25:49,943 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.2 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:25:53,013 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.5 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:25:56,085 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:25:59,161 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:02,231 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:05,309 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:08,380 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:11,451 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:14,523 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:17,596 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:20,668 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:23,731 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:26,793 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:29,861 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:32,913 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:35,963 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:39,014 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:42,066 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:45,123 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:48,178 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:51,229 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:54,286 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:26:57,341 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:00,396 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:03,449 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:06,499 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:09,551 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:12,675 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:15,747 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:18,813 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:21,881 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:24,951 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:28,018 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:31,082 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:34,152 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:37,223 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:40,290 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:43,356 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:46,422 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:49,490 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:52,556 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:55,623 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:27:58,692 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:01,767 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:04,835 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:07,883 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:10,952 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:13,994 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:17,044 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:20,101 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:23,154 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.2 GB of 21 GB virtual memory used
2017-12-07 04:28:26,205 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 9.9 GB of 10 GB physical memory used; 12.6 GB of 21 GB virtual memory used
2017-12-07 04:28:29,257 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 25864 for container-id container_e54_1512469367986_4908_01_000007: 10.2 GB of 10 GB physical memory used; 12.6 GB of 21 GB virtual memory used
2017-12-07 04:28:29,257 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Process tree for container: container_e54_1512469367986_4908_01_000007 has processes older than 1 iteration running over the configured limit. Limit=10737418240, current usage = 10931814400
2017-12-07 04:28:29,257 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Container [pid=25864,containerID=container_e54_1512469367986_4908_01_000007] is running beyond physical memory limits. Current usage: 10.2 GB of 10 GB physical memory used; 12.6 GB of 21 GB virtual memory used. Killing container.
Dump of the process-tree for container_e54_1512469367986_4908_01_000007 :
2017-12-07 04:28:29,258 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from RUNNING to KILLING
2017-12-07 04:28:29,258 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,268 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_e54_1512469367986_4908_01_000007 is : 143
2017-12-07 04:28:29,293 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL
2017-12-07 04:28:29,293 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/b/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,293 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/c/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,293 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/d/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/e/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/f/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/g/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/h/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/i/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/j/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,294 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/k/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/l/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /var/lib/hadoop/data/m/yarn/local/usercache/ebernhardson/appcache/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_e54_1512469367986_4908_01_000007 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE
2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_e54_1512469367986_4908_01_000007 from application application_1512469367986_4908
2017-12-07 04:28:29,295 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_e54_1512469367986_4908_01_000007 for log-aggregation
2017-12-07 04:28:29,295 INFO org.apache.spark.network.yarn.YarnShuffleService: Stopping container container_e54_1512469367986_4908_01_000007
2017-12-07 04:28:30,301 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_e54_1512469367986_4908_01_000007]
2017-12-07 04:28:32,274 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_e54_1512469367986_4908_01_000007
2017-12-07 04:48:37,936 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_e54_1512469367986_4908_01_000007. Current good log dirs are /var/lib/hadoop/data/b/yarn/logs,/var/lib/hadoop/data/c/yarn/logs,/var/lib/hadoop/data/d/yarn/logs,/var/lib/hadoop/data/e/yarn/logs,/var/lib/hadoop/data/f/yarn/logs,/var/lib/hadoop/data/g/yarn/logs,/var/lib/hadoop/data/h/yarn/logs,/var/lib/hadoop/data/i/yarn/logs,/var/lib/hadoop/data/j/yarn/logs,/var/lib/hadoop/data/k/yarn/logs,/var/lib/hadoop/data/l/yarn/logs,/var/lib/hadoop/data/m/yarn/logs
2017-12-07 04:48:37,960 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/lib/hadoop/data/d/yarn/logs/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/stdout
2017-12-07 04:48:37,961 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /var/lib/hadoop/data/d/yarn/logs/application_1512469367986_4908/container_e54_1512469367986_4908_01_000007/stderr