Page MenuHomePhabricator

Hive log4j logging is misconfigured
Closed, ResolvedPublic

Description

Whenever I run a LOAD DATA LOCAL command using the hive command line client (which is currently the only way to load data from a CSV file into Hive), I get the following message before the command runs:

log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender.

From what I can understand, it seems like the DailyRollingFileAppender doesn't have that option, which suggests this code configuring the Hive logging isn't having the desired effect.

Event Timeline

Restricted Application changed the subtype of this task from "Deadline" to "Task". · View Herald TranscriptFeb 15 2019, 10:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
nshahquinn-wmf raised the priority of this task from Medium to Needs Triage.Feb 15 2019, 10:45 PM

FYI, I'm having to think about this again; I'm working on Python wrapper for the Hive command line client as part of T246060, and logspam like this makes it harder to find the important things in the stderr.

Milimetric moved this task from Deprioritized to Operational Excellence on the Analytics board.
Milimetric added subscribers: elukey, Milimetric.

@elukey's going to try and fix this if it's not too full of sharks

Hello Neil, sorry for this lag in following up. I have tested two things:

Stat host (Buster but with hive 1.x deps)

log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender.

Logging initialized using configuration in file:/etc/hive/conf.analytics-hadoop/hive-log4j.properties
WARNING: Hive CLI is deprecated and migration to Beeline is recommended.

New test client running on Buster with hive 2.2.3 deps:

elukey@an-test-client1001:~$ hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-2.3.3.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
hive (default)>

Do you think that this can wait our upgrade to Bigtop? (that includes hive 2.2.3) ?

Do you think that this can wait our upgrade to Bigtop? (that includes hive 2.2.3) ?

Yes, of course. I know you have many bigger things to worry about, so I'm happy with any follow-up at all on this 😊

Neil I think that this can be closed, what do you think?

nshahquinn-wmf claimed this task.

Neil I think that this can be closed, what do you think?

Yes, I think so. Now we have new logspam to worry about, but that's a topic for other tasks 😊