⚓ T291386 Upgrade Refinery Jobs to Spark 3

	Subject	Repo	Branch	Lines +/-
	Bump version to 0.2.0-SNAPSHOT to release 0.2.0	analytics/refinery/source	master	+7 -7

Status	Assigned	Task
Resolved	None	T307500 Airflow Hackathon (May 2022)
Resolved	Ahoelzl	T307505 Refine jobs should be scheduled by Airflow
Resolved	Antoine_Quhen	T356762 [Refine refactoring] Refine jobs should be scheduled by Airflow: implementation
Resolved	None	T291464 Upgrade analytics-hadoop to Spark 3 + scala 2.12
Duplicate	None	T291465 Analytics-test-hadoop Spark3 package upgrade
Duplicate	None	T291466 Analytics-hadoop Spark3 package upgrade (production)
Resolved	JAllemandou	T306955 Spark3 migration - Currently existing airflow jobs
Resolved	Ottomata	T291386 Upgrade Refinery Jobs to Spark 3
Open	None	T209453 Refine: Use Spark SQL instead of Hive JDBC
Open	Ottomata	T307040 Propagate field descriptions from event schemas to Hive event tables and into DataHub
Resolved	Antoine_Quhen	T388793 When doing ADD COLUMN to a struct under a map, Iceberg fails to SELECT it
Resolved	Ottomata	T255818 Refine drops $schema field values
Open	None	T259924 HiveExtensions.convertToSchema does not properly convert arrays of structs
Resolved	tchin	T366487 Event Platform schemas should not support type changes to structs as array element or map value types

Ottomata created this task.Sep 20 2021, 1:56 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 20 2021, 1:56 PM

Ottomata added a subtask: T209453: Refine: Use Spark SQL instead of Hive JDBC .Sep 20 2021, 1:56 PM

Change 656897 had a related patch set uploaded (by Ottomata; author: Joal):

[analytics/refinery/source@master] [WIP] Update to spark-3 and scala-2.12

https://gerrit.wikimedia.org/r/656897

gerritbot added a project: Patch-For-Review.Sep 20 2021, 1:57 PM

• odimitrijevic renamed this task from Upgrade to Spark 3 to Upgrade Refinery Jobs to Spark 3.Sep 20 2021, 4:36 PM

• odimitrijevic triaged this task as Medium priority.

• odimitrijevic updated the task description. (Show Details)

• odimitrijevic added a project: Data-Engineering.

• odimitrijevic moved this task from Incoming to Smart Tools for Better Data on the Analytics board.

Hm, I think this task is also about installing and supporting Spark 3 in favor of Spark 2, with the eventual goal of removing Spark 2. This means making sure everything works with conda and jupyter and airflow.

In T291386#7366295, @Ottomata wrote:

Hm, I think this task is also about installing and supporting Spark 3 in favor of Spark 2, with the eventual goal of removing Spark 2. This means making sure everything works with conda and jupyter and airflow.

We thought it would be better to create a parent task for this one and all you mention. Doing it now.

JAllemandou added a parent task: T291464: Upgrade analytics-hadoop to Spark 3 + scala 2.12.Sep 21 2021, 8:35 AM

JAllemandou mentioned this in T291466: Analytics-hadoop Spark3 package upgrade (production).Sep 21 2021, 8:40 AM

JAllemandou added a parent task: T291466: Analytics-hadoop Spark3 package upgrade (production).

• odimitrijevic moved this task from Incoming (new tickets) to Transform on the Data-Engineering board.Sep 21 2021, 6:42 PM

• odimitrijevic removed a project: Analytics.Jan 12 2022, 12:32 AM

JAllemandou added a parent task: T306955: Spark3 migration - Currently existing airflow jobs.Apr 27 2022, 7:03 AM

Ottomata added a subtask: T255818: Refine drops $schema field values.May 11 2022, 5:41 PM

Ottomata added a subtask: T259924: HiveExtensions.convertToSchema does not properly convert arrays of structs.

Change 798662 had a related patch set uploaded (by Joal; author: Joal):

[analytics/refinery/source@master] Bump version to 0.2.0-SNAPSHOT to release 0.2.0

https://gerrit.wikimedia.org/r/798662