Maniphest T315478

Optimize spark3 conda deb generation
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Antoine_Quhen
	Aug 17 2022, 7:02 PM

Project Tags

Referenced Files

None

Subscribers

Description

In workflow utils when using conda-pack, we should use those 2 parameters:

compress_level to 0 to avoid useless compression of the environment wish is uncompressed later in the process.
n_threads to parallelize the packing process.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		None	T291464 Upgrade analytics-hadoop to Spark 3 + scala 2.12
		Resolved		None	T302819 Replace anaconda-wmf with smaller, non-stacked Conda environments
		Resolved		Antoine_Quhen	T295072 Install spark3 in analytics clusters
		Resolved		xcollazo	T315478 Optimize spark3 conda deb generation

Event Timeline

Antoine_Quhen created this task.Aug 17 2022, 7:02 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 17 2022, 7:02 PM

Antoine_Quhen added a parent task: T295072: Install spark3 in analytics clusters.Aug 17 2022, 7:02 PM

• EChetty moved this task from Backlog to Next Up (revisit every 2 sprints) on the Data Pipelines board.Aug 19 2022, 11:22 AM

• EChetty edited projects, added: Data-Engineering-Planning; removed: Data-Engineering.Sep 6 2022, 10:40 AM

• EChetty moved this task from Backlog to Pipelines on the Data-Engineering-Planning board.Sep 6 2022, 10:46 AM

In T321736, we moved to a pure Dockerfile approach, and we now do this optimization here: https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/blob/eb2ba59fa699e78625b371c43ac62294a9167d0f/docker/Dockerfile#L64-71.

We can use this as reference for other conda based debian packages.