I am testing webrequest_load in Hadoop test, using the following configs:
- refinery deployed on hdfs following our regular deployment procedure
- ad-hoc /user/elukey/oozie directory with my changes
- https://gerrit.wikimedia.org/r/c/analytics/refinery/+/491791 as test bundle
Everything works as expected, except the yarn refine job that throws this error:
Error: Error running query: java.lang.NoSuchMethodError: com.maxmind.geoip2.DatabaseReader.<init>(Lcom/maxmind/geoip2/DatabaseReader$Builder;Lcom/maxmind/geoip2/DatabaseR
eader$1;)V (state=,code=0)
org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.NoSuchMethodError: com.maxmind.geoip2.DatabaseReader.<init>(Lcom/maxmind/geoip2/DatabaseReade
r$Builder;Lcom/maxmind/geoip2/DatabaseReader$1;)V
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:313)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
at org.apache.hive.beeline.Commands.executeInternal(Commands.java:997)
at org.apache.hive.beeline.Commands.execute(Commands.java:1205)
at org.apache.hive.beeline.Commands.sql(Commands.java:1134)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1314)
at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1178)
at org.apache.hive.beeline.BeeLine.executeFile(BeeLine.java:1155)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1026)
at org.apache.oozie.action.hadoop.Hive2Main.runBeeline(Hive2Main.java:267)
at org.apache.oozie.action.hadoop.Hive2Main.run(Hive2Main.java:241)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java
[..]The above seems very close to what described in https://stackoverflow.com/questions/56261083/how-to-fix-java-lang-nosuchmethoderror-for-geoip2-java-running-under-hive
In Hadoop test we are running Bigtop with Hive 2.x, that is different from what what we run in production. My previous tests were all based on old versions of refinery, there might be something to do before upgrading to Bigtop.
It could also be that it is a problem with my testing environment :)