Maniphest T291464

Upgrade analytics-hadoop to Spark 3 + scala 2.12
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	JAllemandou
	Sep 21 2021, 8:35 AM

Tags

Referenced Files

None

Subscribers

Description

This is the parent task about upgrading the analytics hadoop cluster to using Spark 3 with Scala 2.12. Actual work should be listed in sub-tasks.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		None	T307500 Airflow Hackathon (May 2022)
		Resolved		Ahoelzl	T307505 Refine jobs should be scheduled by Airflow
		Resolved		Antoine_Quhen	T356762 [Refine refactoring] Refine jobs should be scheduled by Airflow: implementation
		Resolved		None	T291464 Upgrade analytics-hadoop to Spark 3 + scala 2.12
		Duplicate		None	T291465 Analytics-test-hadoop Spark3 package upgrade
		Duplicate		None	T291466 Analytics-hadoop Spark3 package upgrade (production)
		Resolved		Antoine_Quhen	T295072 Install spark3 in analytics clusters
		Resolved		Antoine_Quhen	T309227 Create conda-base-env with last pyspark
		Resolved		Antoine_Quhen	T312882 Puppetize Spark 3 installation using conda-analytics env
		Resolved		Antoine_Quhen	T310578 Build and install spark3 assembly
		Resolved		xcollazo	T315478 Optimize spark3 conda deb generation
		Resolved		JAllemandou	T306955 Spark3 migration - Currently existing airflow jobs
		Resolved		Ottomata	T291386 Upgrade Refinery Jobs to Spark 3
		Open		None	T209453 Refine: Use Spark SQL instead of Hive JDBC
		Open		Ottomata	T307040 Propagate field descriptions from event schemas to Hive event tables and into DataHub
		Resolved		Antoine_Quhen	T388793 When doing ADD COLUMN to a struct under a map, Iceberg fails to SELECT it
		Resolved		Ottomata	T255818 Refine drops $schema field values
		Open		None	T259924 HiveExtensions.convertToSchema does not properly convert arrays of structs
		Resolved		tchin	T366487 Event Platform schemas should not support type changes to structs as array element or map value types
		Resolved		Antoine_Quhen	T308168 [POC] Use airflow-installed Spark3 for an Airflow job

Event Timeline

JAllemandou created this task.Sep 21 2021, 8:35 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 21 2021, 8:35 AM

JAllemandou added a subtask: T291386: Upgrade Refinery Jobs to Spark 3.Sep 21 2021, 8:35 AM

JAllemandou added a subtask: T209453: Refine: Use Spark SQL instead of Hive JDBC .

JAllemandou added a subtask: T291466: Analytics-hadoop Spark3 package upgrade (production).Sep 21 2021, 8:41 AM

• odimitrijevic added a project: Data-Engineering.Sep 23 2021, 4:33 PM

• odimitrijevic triaged this task as Medium priority.Sep 30 2021, 5:20 PM

• odimitrijevic moved this task from Incoming to Smart Tools for Better Data on the Analytics board.

nshahquinn-wmf subscribed.Oct 18 2021, 9:23 PM

Milimetric closed subtask T291465: Analytics-test-hadoop Spark3 package upgrade as Declined.Oct 28 2021, 9:47 PM

Milimetric mentioned this in T291465: Analytics-test-hadoop Spark3 package upgrade.

Milimetric reopened subtask T291465: Analytics-test-hadoop Spark3 package upgrade as Open.Oct 28 2021, 9:50 PM

• odimitrijevic added a project: Epic.Nov 1 2021, 4:46 PM

• odimitrijevic moved this task from Incoming (new tickets) to Tag with Icebox on the Data-Engineering board.

Ottomata added a subtask: T295072: Install spark3 in analytics clusters.Nov 12 2021, 9:51 PM

• odimitrijevic mentioned this in T298666: Prometheus metrics for Spark 3.Jan 6 2022, 1:43 AM

JAllemandou added a subtask: T306955: Spark3 migration - Currently existing airflow jobs.Apr 27 2022, 6:55 AM

JAllemandou added a subtask: T308168: [POC] Use airflow-installed Spark3 for an Airflow job.May 12 2022, 9:54 AM

JArguello-WMF closed subtask T308168: [POC] Use airflow-installed Spark3 for an Airflow job as Resolved.May 31 2022, 3:24 PM

JArguello-WMF removed a project: Analytics.Jun 6 2022, 5:39 PM

JArguello-WMF closed subtask T306955: Spark3 migration - Currently existing airflow jobs as Resolved.Jul 5 2022, 3:42 PM

BTullis subscribed.Jul 5 2022, 3:55 PM

xcollazo closed subtask T295072: Install spark3 in analytics clusters as Resolved.Nov 4 2022, 2:29 PM

JArguello-WMF moved this task from Tag with Icebox to Event Platform Backlog on the Data-Engineering board.Jun 29 2023, 11:03 PM

Can this task be closed now?

I think it can be closed, yes - the remaining tasks are almost all about Refine improvements we can now implement thanks to the move to Spark3.

Ahoelzl moved this task from Event Platform Backlog to Backlog on the Data-Engineering board.Jan 18 2025, 12:02 AM

Ahoelzl moved this task from Backlog to Tag with Roadmap on the Data-Engineering board.Mar 12 2025, 4:41 PM

Ahoelzl edited projects, added Data-Engineering-Roadmap; removed Data-Engineering.

Ottomata closed subtask T291386: Upgrade Refinery Jobs to Spark 3 as Resolved.Apr 28 2025, 12:53 PM

nshahquinn-wmf closed this task as Resolved.Jul 1 2025, 2:18 AM