Page MenuHomePhabricator
Create Task
Maniphest T159962

Spark 2 as cluster default (working with oozie)
Closed, ResolvedPublic13 Estimated Story Points
Actions

Description

Here is the plan I suggest to globally move production to Spark 2:

@Ottomata : Comments welcome!

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 DuplicateNoneT159961 Unlock Spark with Oozie
ResolvedJAllemandouT159962 Spark 2 as cluster default (working with oozie)

Event Timeline

Ottomata created this task.Mar 8 2017, 5:47 PM

Can we do it? How hard is it?

• Nuria triaged this task as Low priority.Mar 13 2017, 3:45 PM
• Nuria moved this task from Incoming to Wikistats on the Analytics board.
Ottomata added a comment.Mar 14 2017, 3:31 PM

Need a place to park some notes:

spark build your own deb

export JAVA_HOME
- add openjdk into build depends
- export http proxies
- alter .m2/settings.xml to add proxy

<settings>
<proxies>
   <proxy>
      <id>http-wikimedia</id>
      <active>true</active>
      <protocol>http</protocol>
      <host>webproxy.eqiad.wmnet</host>
      <port>8080</port>
    </proxy>
    <proxy>
       <id>https-wikimedia</id>
       <active>true</active>
       <protocol>https</protocol>
       <host>webproxy.eqiad.wmnet</host>
       <port>8080</port>
     </proxy>
  </proxies>

</settings>

add --settings=/path/to/settings.xml to mvn command (BUILD_OPTS in  debian/do-component-build) ???





$MVN  help:evaluate -Dexpression=project.version --skip-java-test -Dcdh.build=true -Divy.home=/tmp/buildd/.ivy2 -Dsbt.ivy.home=/tmp/buildd/.ivy2 -Duser.home=/tmp/buildd -Drepo.maven.org= -Dreactor.repo=file:///tmp/buildd/.m2/repository -DskipTests -DrecompileMode=all --settings=./debian/m2-settings.xml

make-distribution.sh
 --skip-java-test and --with-tachyon --tgz need to be shifted???
 
 need to add -Pyarn to build with yarn deps???
 
 
 -Pyarn -Phadoop-2.6 -Phive -Phive-thriftserver ???
• Nuria moved this task from Wikistats to Deprioritized on the Analytics board.Jun 15 2017, 4:02 PM
JAllemandou renamed this task from Spike: Spark 2.x as cluster default (working with oozie) to Spark 2.x as cluster default (working with oozie).Feb 22 2018, 10:24 AM

Can we try that: https://community.hortonworks.com/questions/114243/oozie-spark2-compatibility.html ??

JAllemandou renamed this task from Spark 2.x as cluster default (working with oozie) to Spark 2.2.1 as cluster default (working with oozie).Feb 22 2018, 10:25 AM
JAllemandou claimed this task.
JAllemandou edited projects, added Analytics-Kanban; removed Analytics.
JAllemandou moved this task from Next Up to In Progress on the Analytics-Kanban board.
JAllemandou set the point value for this task to 8.
Ottomata added a comment.Feb 22 2018, 7:33 PM

TODO: puppetize oozie with spark 2.2.1: like https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_spark-component-guide/content/ch_oozie-spark-action.html#spark-config-oozie-spark2

JAllemandou updated the task description. (Show Details)Feb 23 2018, 8:12 AM
JAllemandou changed the point value for this task from 8 to 13.
JAllemandou updated the task description. (Show Details)Feb 28 2018, 4:41 PM
gerritbot subscribed.Feb 28 2018, 10:15 PM

Change 415465 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Automate installation of spark2 oozie sharelib

https://gerrit.wikimedia.org/r/415465

gerritbot added a project: Patch-For-Review.Feb 28 2018, 10:15 PM
gerritbot added a comment.Mar 1 2018, 4:28 PM

Change 415584 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Temporarliy run banner impression spark streaming job from 2.2.1 .jar

https://gerrit.wikimedia.org/r/415584

gerritbot added a comment.Mar 1 2018, 4:29 PM

Change 415584 merged by Ottomata:
[operations/puppet@production] Temporarliy run banner impression spark streaming job from 2.2.1 .jar

https://gerrit.wikimedia.org/r/415584

Ottomata updated the task description. (Show Details)Mar 1 2018, 4:36 PM
gerritbot added a comment.Mar 1 2018, 5:17 PM

Change 415465 merged by Ottomata:
[operations/puppet@production] Automate installation of spark2 oozie sharelib

https://gerrit.wikimedia.org/r/415465

gerritbot added a comment.Mar 1 2018, 5:25 PM

Change 415602 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Properly install spark2_oozie_sharelib_install.sh

https://gerrit.wikimedia.org/r/415602

gerritbot added a comment.Mar 1 2018, 7:18 PM

Change 415602 merged by Ottomata:
[operations/puppet@production] Properly install spark2_oozie_sharelib_install.sh

https://gerrit.wikimedia.org/r/415602

Ottomata updated the task description. (Show Details)Mar 1 2018, 7:25 PM

@joal, spark2.2.1 sharelib exists. Let me know if I can remove our test spark2_test0 one.

gerritbot added a comment.Mar 1 2018, 7:29 PM

Change 415634 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Also copy in hive-site.xml to spark2 oozie sharelib

https://gerrit.wikimedia.org/r/415634

gerritbot added a comment.Mar 1 2018, 7:30 PM

Change 415634 merged by Ottomata:
[operations/puppet@production] Also copy in hive-site.xml to spark2 oozie sharelib

https://gerrit.wikimedia.org/r/415634

Ottomata added a subscriber: mforns.Mar 1 2018, 7:34 PM

@mforns FYI, we'd like to get your Sanitize job merged before we proceed with this...and we're hoping we can do this next week! :D

JAllemandou added a comment.Mar 2 2018, 8:31 AM

@Ottomata : I tested the spark2.2.1 sharelib, and it failed. I think this is the issue:

hdfs dfs -ls /user/oozie/share/lib/lib_20170228165236/spark2.2.1 | grep oozie-

hdfs dfs -ls /user/oozie/share/lib/lib_20170228165236/spark2_test0 | grep oozie-
-rw-r--r--   3 oozie hadoop      26013 2018-02-22 18:57 /user/oozie/share/lib/lib_20170228165236/spark2_test0/oozie-sharelib-spark-4.1.0-cdh5.10.0.jar
-rw-r--r--   3 oozie hadoop      26013 2018-02-22 18:57 /user/oozie/share/lib/lib_20170228165236/spark2_test0/oozie-sharelib-spark.jar
gerritbot added a comment.Mar 2 2018, 9:04 AM

Change 415812 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery/source@master] Update spark jobs to use hive context

https://gerrit.wikimedia.org/r/415812

gerritbot added a comment.Mar 6 2018, 4:57 PM

Change 416713 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix spark2 oozie sharelib install command

https://gerrit.wikimedia.org/r/416713

Ottomata added a comment.Mar 6 2018, 4:57 PM

Oof, joal, ya, copied spark-assembly instead of oozie-sharelib. Fixed now.

gerritbot added a comment.Mar 6 2018, 5:01 PM

Change 416713 merged by Ottomata:
[operations/puppet@production] Fix spark2 oozie sharelib install command

https://gerrit.wikimedia.org/r/416713

JAllemandou moved this task from In Progress to In Code Review on the Analytics-Kanban board.Mar 9 2018, 8:17 PM
Ottomata renamed this task from Spark 2.2.1 as cluster default (working with oozie) to Spark 2 as cluster default (working with oozie).Apr 5 2018, 4:26 PM
Ottomata updated the task description. (Show Details)
gerritbot added a comment.Apr 5 2018, 7:19 PM

Change 424380 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/debs/spark2@debian] 2.3.0 Hadoop 2.6 release

https://gerrit.wikimedia.org/r/424380

gerritbot added a comment.Apr 5 2018, 7:21 PM

Change 424380 merged by Ottomata:
[operations/debs/spark2@debian] 2.3.0 Hadoop 2.6 release

https://gerrit.wikimedia.org/r/424380

Ottomata updated the task description. (Show Details)Apr 5 2018, 7:35 PM

Spark 2.3 installed fleet wide.

I also updated the spark2 spark-assemply.zip file and added a new oozie sharelib spark2.3.0:

sudo -u spark hdfs dfs -put /usr/lib/spark2/spark2-assembly.zip hdfs:///user/spark/share/lib/spark2-assembly.zip
sudo -u oozie oozie admin -oozie $OOZIE_URL -shareliblist | grep spark
spark
spark2_test0
spark2.2.1
spark2.3.0
gerritbot added a comment.Apr 5 2018, 8:24 PM

Change 424444 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/debs/spark2@debian] Install spark2-thriftserver executable

https://gerrit.wikimedia.org/r/424444

gerritbot added a comment.Apr 5 2018, 8:25 PM

Change 424444 merged by Ottomata:
[operations/debs/spark2@debian] Install spark2-thriftserver executable

https://gerrit.wikimedia.org/r/424444

gerritbot added a comment.Apr 6 2018, 2:00 PM

Change 424593 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Install the Spark 2 yarn shuffle service jar over Spark 1's

https://gerrit.wikimedia.org/r/424593

Ottomata updated the task description. (Show Details)Apr 6 2018, 2:01 PM
gerritbot added a comment.Apr 9 2018, 7:12 PM

Change 425084 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery/source@master] Add HiveServer to spark-refine for schema changes

https://gerrit.wikimedia.org/r/425084

gerritbot added a comment.Apr 10 2018, 3:43 PM

Change 425289 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use spark2 for Refine job

https://gerrit.wikimedia.org/r/425289

gerritbot added a comment.Apr 10 2018, 4:24 PM

Change 425306 had a related patch set uploaded (by Ottomata; owner: Joal):
[analytics/refinery/source@master] Add HiveServer to spark-refine for schema changes

https://gerrit.wikimedia.org/r/425306

gerritbot added a comment.Apr 10 2018, 4:25 PM

Change 425306 merged by Ottomata:
[analytics/refinery/source@master] Add HiveServer to spark-refine for schema changes

https://gerrit.wikimedia.org/r/425306

gerritbot added a comment.Apr 10 2018, 4:25 PM

Change 425084 abandoned by Joal:
Add HiveServer to spark-refine for schema changes

Reason:
Cherry picked in another change

https://gerrit.wikimedia.org/r/425084

gerritbot added a comment.Apr 10 2018, 4:28 PM

Change 415812 merged by Ottomata:
[analytics/refinery/source@master] Update spark jobs to use hive context

https://gerrit.wikimedia.org/r/415812

gerritbot added a comment.Apr 10 2018, 6:04 PM

Change 424593 merged by Ottomata:
[operations/puppet@production] Install the Spark 2 yarn shuffle service jar over Spark 1's

https://gerrit.wikimedia.org/r/424593

Stashbot subscribed.Apr 10 2018, 6:18 PM

Mentioned in SAL (#wikimedia-analytics) [2018-04-10T18:18:54Z] <ottomata> restarting all hadoop nodemanagers, 3 at a time to pick up spark2-yarn-shuffle.jar T159962

gerritbot added a comment.Apr 10 2018, 8:05 PM

Change 425347 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] Refine - Don't call sys.exit if running in YARN

https://gerrit.wikimedia.org/r/425347

gerritbot added a comment.Apr 10 2018, 8:34 PM

Change 425289 merged by Ottomata:
[operations/puppet@production] Use spark2 for Refine job and banner-streaming job

https://gerrit.wikimedia.org/r/425289

gerritbot added a comment.Apr 10 2018, 8:39 PM

Change 425347 merged by Ottomata:
[analytics/refinery/source@master] Refine - Don't call sys.exit if running in YARN

https://gerrit.wikimedia.org/r/425347

gerritbot added a comment.Apr 11 2018, 6:00 PM

Change 425578 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] DataFrameToHive - Use DataFrame .write.parquet instead of .insertInto

https://gerrit.wikimedia.org/r/425578

gerritbot added a comment.Apr 11 2018, 7:39 PM

Change 425597 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] RefineTarget - Use Hadoop FS to infer input format rather than Spark

https://gerrit.wikimedia.org/r/425597

gerritbot added a comment.Apr 12 2018, 12:23 AM

Change 425578 merged by Ottomata:
[analytics/refinery/source@master] DataFrameToHive - Use DataFrame .write.parquet instead of .insertInto

https://gerrit.wikimedia.org/r/425578

gerritbot added a comment.Apr 12 2018, 12:25 AM

Change 425597 merged by Ottomata:
[analytics/refinery/source@master] RefineTarget - Use Hadoop FS to infer input format rather than Spark

https://gerrit.wikimedia.org/r/425597

Ottomata updated the task description. (Show Details)Apr 12 2018, 6:48 PM
JAllemandou moved this task from In Code Review to Done on the Analytics-Kanban board.Apr 16 2018, 7:04 AM
gerritbot added a comment.Apr 16 2018, 3:04 PM

Change 426943 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Point refine job at 0.0.62 jar version

https://gerrit.wikimedia.org/r/426943

gerritbot added a comment.Apr 16 2018, 3:09 PM

Change 426943 merged by Ottomata:
[operations/puppet@production] Point refine job at 0.0.62 jar version

https://gerrit.wikimedia.org/r/426943

• Nuria closed this task as Resolved.Apr 17 2018, 9:24 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits