Page MenuHomePhabricator
Create Task
Maniphest T336513

Figure out a way to automatize deployment of the spark assembly file
Open, MediumPublic
Actions

Description

(Related to T335721)

Yarn requires the definition of spark.yarn.archive, a zip or jar file containing all the jars from Spark. This is needed so that we don't loose time at the beginning of every Spark job uploading all of its jars.

We have a manual solution for Spark3 at https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/blob/main/generate_spark_assembly.sh.

We also have an automated solution for this for Spark2 here. However, this solution does not work with our new way of deploying Spark via pyspark.

In this task we should reconcile these two approaches and make it automated for Spark3 as well.

Related Objects
Search...

Event Timeline

xcollazo created this task.May 11 2023, 3:10 PM
xcollazo mentioned this in T335721: Add support for Iceberg in Spark.May 11 2023, 3:13 PM
JArguello-WMF added a project: Data Engineering and Event Platform Team.Jun 30 2023, 5:52 PM
xcollazo mentioned this in T344910: Deploy additional yarn shuffler services to support several versions of spark in parallel.Oct 26 2023, 6:15 PM
Ottomata subscribed.Oct 27 2023, 1:38 PM

BTW, we should probably clean out the old puppet code that did this, if we aren't using it anymore. E.g. https://github.com/wikimedia/operations-puppet/blob/production/modules/profile/files/hadoop/spark3/spark3_upload_assembly.sh

xcollazo added a comment.Oct 27 2023, 2:54 PM

An easy fix for this is to just delete all assembly files, and remove spark.yarn.archive from our spark config.

The con from this approach is that the assembly will be zipped and shipped to the cluster on every spark job. I do wonder how long this takes... perhaps just a couple seconds?

BTullis added a project: Data-Platform.Nov 6 2023, 9:52 AM
BTullis added a project: Data-Platform-SRE.
• lbowmaker moved this task from Incoming to SRE on the Data-Platform board.Nov 10 2023, 2:47 PM
• lbowmaker edited projects, added Data-Engineering; removed Data Engineering and Event Platform Team.
Gehel triaged this task as Medium priority.Nov 15 2023, 9:38 AM
Gehel moved this task from Incoming to Ready for Work on the Data-Platform-SRE board.
Gehel moved this task from Ready for Work to Quarterly Goals on the Data-Platform-SRE board.Nov 15 2023, 9:43 AM
• lbowmaker moved this task from Incoming (new tickets) to Radar (External Teams) on the Data-Engineering board.Feb 12 2024, 2:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits