Page MenuHomePhabricator
Create Task
Maniphest T126824

Make Pyspark work on Altiscale cluster
Closed, ResolvedPublic
Actions

Description

[kjschiroo@desktop2-ia Scripts]$ /opt/alti-spark-1.4.1.hadoop24.hive13/bin/spark-submit --master yarn persistent_words.py 
Hive history file=/home/hive/log/kjschiroo/hive_job_log_a2b731dd-36cc-4c78-99f1-fec369cf753b_1746474642.txt
Hive history file=/home/hive/log/kjschiroo/hive_job_log_ad1ffe44-2538-4db9-83d2-8dc267f8fd87_1680284182.txt
Traceback (most recent call last):
  File "/home/kjschiroo/WikiEd/WikiEd_Altiscale/Scripts/persistent_words.py", line 29, in <module>
    get_persistent_words_for_users(user_list[0:10])
  File "/home/kjschiroo/WikiEd/WikiEd_Altiscale/Scripts/persistent_words.py", line 20, in get_persistent_words_for_users
    results = sqlContext.sql(query).collect()
  File "/opt/alti-spark-1.4.1.hadoop24.hive13/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 281, in collect
  File "/opt/alti-spark-1.4.1.hadoop24.hive13/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
  File "/opt/alti-spark-1.4.1.hadoop24.hive13/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:190)
    at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:203)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:217)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1781)
    at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:885)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:286)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:884)
    at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:378)
    at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:259)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 57 more
Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
    at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
    at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
    at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
    ... 62 more
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
    at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
    ... 64 more

persistent_words.py

import users
from pyspark.sql import HiveContext
from pyspark import SparkContext

sc = SparkContext()
sqlContext = HiveContext(sc)

def get_persistent_words_for_users(users):
    template = """
               SELECT *
               FROM halfak.word_persistence_enwiki_20150602
               WHERE {user_criteria}
               """
    user_str = _assemble_or_equals_string("user_text", users)
    query = template.format(user_criteria=user_str)
    results = sqlContext.sql(query).collect()

def _assemble_or_equals_string(field_name, or_list):
    single_criteria = ['{0}="{1}"'.format(field_name, entry) for entry in or_list]
    joined = " OR ".join(single_criteria)
    return joined

user_list = users.load_users("fall_2014_students.csv")
get_persistent_words_for_users(user_list[0:10])

Event Timeline

Kjschiroo created this task.Feb 12 2016, 11:08 PM
Kjschiroo raised the priority of this task from to Needs Triage.
Kjschiroo updated the task description. (Show Details)
Kjschiroo added projects: Research, Measuring-value-added.
Kjschiroo moved this task to Backlog on the Research board.
Kjschiroo added subscribers: Kjschiroo, JAllemandou.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 12 2016, 11:08 PM
Kjschiroo moved this task from Backlog to In Progress on the Research board.Feb 12 2016, 11:10 PM

@JAllemandou, could you take a look at this?

JAllemandou added a comment.Feb 13 2016, 6:35 AM

I lack the users package to run your code, but you can try launching spark-submit with that option:

--driver-class-path $(find /opt/hadoop/share/hadoop/mapreduce/lib/hadoop-lzo-* | head -n 1)
Kjschiroo added a comment.Feb 16 2016, 4:27 PM

I'm not sure if that fixed it, but it definitely changed it. What is it doing?

JAllemandou added a comment.Feb 16 2016, 5:09 PM

This finds the hadoop-lzo-* jar file in hadoop lib, and adds it to the spark driver (not there by default).

Kjschiroo moved this task from In Progress to Done (current quarter) on the Research board.Mar 2 2016, 1:44 AM
• ggellerman closed this task as Resolved.Apr 29 2016, 10:14 PM
• ggellerman claimed this task.
• ggellerman edited projects, added Research-Archive; removed Research.
• DarTar moved this task from Default to Q1-Q4-FY17 on the Research-Archive board.Jan 9 2018, 12:00 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits