Page MenuHomePhabricator
Create Task
Maniphest T283126

Failures registered by drop_event on an-launcher1002
Closed, ResolvedPublic
Actions

Description

The drop_event unit on an-launcher1002 has been failing a couple of times for:

NoViableAltException(340@[213:1: tableName : (db= identifier DOT tab= identifier -> ^( TOK_TABNAME $db $tab) |tab= identifier -> ^( TOK_TABNAME $tab) );])
	at org.antlr.runtime.DFA.noViableAlt(DFA.java:158)
	at org.antlr.runtime.DFA.predict(DFA.java:116)
	at org.apache.hadoop.hive.ql.parse.HiveParser_FromClauseParser.tableName(HiveParser_FromClauseParser.java:5041)
	at org.apache.hadoop.hive.ql.parse.HiveParser.tableName(HiveParser.java:41882)
	at org.apache.hadoop.hive.ql.parse.HiveParser.showStatement(HiveParser.java:17678)
	at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:3878)
	at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2382)
	at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1333)
	at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:208)
	at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:77)
	at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:70)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
FAILED: ParseException line 1:17 cannot recognize input near '2021' '-' '05' in table name
")

Details

SubjectRepoBranchLines +/-
drop_event job - Use dedicated Hive CLI log fileoperations/puppetproduction+7 -43
Hive log4j -= Use PidDailyRollingFileAppender for CLI sessionsoperations/puppetproduction+2 -3
Customize query in gerrit

Related Objects

Event Timeline

elukey created this task.May 19 2021, 6:12 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 19 2021, 6:12 AM
Ottomata added a comment.May 19 2021, 1:42 PM

I'll paste what I wrote in analytics-alerts in response to this:

I reran this manually last night with no issue. It looks like somehow a bad table name is making it into the list of tables from which the refinery-drop-older-than script is trying to drop partitions for.

I suspect that for some reason the refinery python hive.tables() function is failing due to some logging issue, but the command returns successfully so the bad output is making into return value of the list of tables. I can't reproduce this so I don't know exactly what is going on. Let's keep watching.

Ottomata claimed this task.May 19 2021, 1:43 PM
Ottomata added a project: Analytics-Kanban.
odimitrijevic moved this task from Incoming to Operational Excellence on the Analytics board.May 20 2021, 5:09 PM
Stashbot added a comment.May 24 2021, 3:19 PM

Mentioned in SAL (#wikimedia-analytics) [2021-05-24T15:19:10Z] <ottomata> rm -rf /tmp/analytics/* on an-launcher1002 - T283126

Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.May 26 2021, 10:15 PM
Ottomata added a comment.Jun 2 2021, 3:40 PM

So, bad output contains an error like:

ERROR Unable to move file /tmp/analytics/hive.log.2021-06-01 to /tmp/analytics/hive.log.2021-06-01

Looking at that file, I see logs totally unrelated to drop_event (they look more like some reportupdater hive job logs). I think what is happening is that concurrent Hive CLI sessions are running at the same time on an-launcher1002, and the log4j using DailyRollingFileAppender is trying to work with the same log file from the different Hive CLI sessions.

The hive-log4j.conf file we have even has the comment:

# Use the PidDailyerRollingFileAppend class instead if you want to use separate log files
# for different CLI session.

Let's try PidDailyerRollingFileAppend.

gerritbot added a comment.Jun 2 2021, 3:41 PM

Change 697802 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Hive log4j -= Use PidDailyRollingFileAppender for CLI sessions

https://gerrit.wikimedia.org/r/697802

gerritbot added a project: Patch-For-Review.Jun 2 2021, 3:41 PM

Change 697802 merged by Ottomata:

[operations/puppet@production] Hive log4j -= Use PidDailyRollingFileAppender for CLI sessions

https://gerrit.wikimedia.org/r/697802

Ottomata moved this task from In Progress to In Code Review on the Analytics-Kanban board.Jun 2 2021, 3:58 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 2 2021, 4:10 PM
gerritbot added a comment.Jun 7 2021, 2:13 PM

Change 698519 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] drop_event job - Use dedicated Hive CLI log file

https://gerrit.wikimedia.org/r/698519

gerritbot added a project: Patch-For-Review.Jun 7 2021, 2:13 PM
gerritbot added a comment.Jun 7 2021, 2:20 PM

Change 698519 merged by Ottomata:

[operations/puppet@production] drop_event job - Use dedicated Hive CLI log file

https://gerrit.wikimedia.org/r/698519

Maintenance_bot removed a project: Patch-For-Review.Jun 7 2021, 3:10 PM
Ottomata moved this task from In Code Review to Done on the Analytics-Kanban board.Jun 9 2021, 2:58 PM
Ottomata closed this task as Resolved.Jun 9 2021, 5:23 PM
BTullis mentioned this in T279304: Incorrect log4j configuration on hive servers causing excessive logging.Jul 28 2021, 1:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL