Change Details

Spark 2+ is a real improvelent over 1.6, it'd be great if we could have it available, and gently move our jobs to the new APIs.

Spark 2+ is a real improvelent over 1.6, it'd be great if we could have it available, and gently move our jobs to the new APIs. Loose end TODOs: [x] remove spark2-beeline [x] spark-sql logging is too verbose with provided log4j.properties [x] Make spark2 use hadoop native libs [x] Make a spark2 assembly jar and put hdfs [] Wikitech documentation [] email announcement

Spark 2+ is a real improvelent over 1.6, it'd be great if we could have it available, and gently move our jobs to the new APIs. Loose end TODOs: [x] remove spark2-beeline [x] spark-sql logging is too verbose with provided log4j.properties [x] Make spark2 use hadoop native libs [x] Make a spark2 assembly jar and put hdfs [] Wikitech documentation [] email announcement