Running spark on jupyter in SWAP
ActivePublic
Actions

Authored by EBernhardson on Nov 22 2017, 7:32 PM.

Tags

None

Referenced Files

	F10942756:
	Nov 22 2017, 7:32 PM

Subscribers

None

	!pip install findspark
	import os
	import findspark
	os.environ['SPARK_HOME'] = '/usr/lib/spark2';
	findspark.init()
	import pyspark
	import pyspark.sql
	conf = pyspark.SparkConf()
	sc = pyspark.SparkContext(conf=conf)
	sqlContext = pyspark.sql.HiveContext(sc)
	df = sqlContext.read.parquet('hdfs://analytics-hadoop/user/ebernhardson/mjolnir/20171023')
	df.show()

EBernhardson changed the title of this paste from untitled to Running spark on jupyter in SWAP.Nov 22 2017, 8:57 PM

Running spark on jupyter in SWAPActivePublicActions