In order, one of the following:
Keeping the other 2 options for the record:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | JAllemandou | T168550 Make tranquility work with Spark | |||
| Resolved | Ottomata | T158334 Make Spark 2.1 easily available on new CDH5.10 cluster |
For debianization, we could model off what we did for druid:
https://github.com/wikimedia/operations-debs-druid/blob/debian/debian/README.Debian
Hm! I think we may be able to push realtime stats to druid with the Tranquility API. It's got a Spark interface:
https://github.com/druid-io/tranquility/blob/master/docs/spark.md
Change 370865 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Allow ANALYTICS_NETWORKS to talk to druid zookeeper cluster
Change 370865 merged by Ottomata:
[operations/puppet@production] Allow ANALYTICS_NETWORKS to talk to druid zookeeper cluster
Yargh. I keep getting stuck on Jackson databind dependency issues. This is the same issue that druid has when running Hadoop indexing task, that somehow we've gotten over by setting -Dhadoop.mapreduce.job.user.classpath.first=true in the middlemanager runtime.properties. This seems to work ok when it is just Druid and Hadoop doing stuff, but adding Spark into the mix makes things even more difficult. I get
java.lang.VerifyError: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides final method deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
I believe this is because spark executors are auto loading .jars out of various Hadoop class paths, e.g. /usr/lib/hadoop-mapreduce/jackson-databind-2.2.3.jar, and those are taking precidence over the newer jackson-databind 2.4.6 version that Druid and also refinery-job need. I've tried all kinds of ways (spark.{driver,executor}.extraJavaOptions, spark.{driver,executor}.userClassPathFirst, etc.) of getting Spark to use 2.4.6, but without much luck.
Anyway, I'll park the modified version of Joseph's BannerImpressionsStream job here for now. Maybe Joseph and I can figure this out together some other time.
https://gist.github.com/ottomata/5020480e15eae6fe80c54e2aa4c80b78
Change 375762 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production] Update pivot config with explicit dimensions
Change 375762 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production] Update pivot config with explicit dimensions
Change 375762 merged by Elukey:
[operations/puppet@production] Update pivot config with explicit dimensions