(Note: This is not an urgent/important request but I was talking to @elukey on IRC and so decided to document this here.)
Hi. I usually run a Hive query from Spark and then while it executes, I move on to something else and come back to check the result after a while. This is especially true for queries that take some time to finish executing.
But after the query has finished and the result is displayed, I see some logging messages, such as the one below that make it difficult to find the output result:
21/02/16 17:19:22 INFO DAGScheduler: Executor lost: 92 (epoch 2) 21/02/16 17:19:22 INFO BlockManagerMasterEndpoint: Trying to remove executor 92 from BlockManagerMaster. 21/02/16 17:19:22 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(92, analytics1070.eqiad.wmnet, 38763, None) 21/02/16 17:19:22 INFO BlockManagerMaster: Removed 92 successfully in removeExecutor 21/02/16 17:19:22 INFO YarnScheduler: Executor 92 on analytics1070.eqiad.wmnet killed by driver. 21/02/16 17:19:22 INFO ExecutorAllocationManager: Existing executor 92 has been removed (new total is 0)
As well as cleanup messages,
21/02/16 17:25:19 INFO ContextCleaner: Cleaned accumulator 76 21/02/16 17:25:19 INFO ContextCleaner: Cleaned accumulator 142 21/02/16 17:25:19 INFO ContextCleaner: Cleaned accumulator 200 21/02/16 17:25:19 INFO ContextCleaner: Cleaned accumulator 192 21/02/16 17:25:19 INFO ContextCleaner: Cleaned accumulator 81 21/02/16 17:25:19 INFO ContextCleaner: Cleaned accumulator 164 21/02/16 17:25:19 INFO ContextCleaner: Cleaned accumulator 155
These are repeated for many lines and I have noticed they don't happen immediately after the query finishes but after some idling.
Is there a way to suppress/filter these messages so that I can just observe the result and not worry about losing it in the backlog? Thanks!