Analytics-test-hadoop Spark3 package upgrade
Closed, DuplicatePublic
Actions

Assigned To

None

Authored By

	JAllemandou
	Sep 21 2021, 8:39 AM

Description

Decide whether we create the package, or reuse a bigtop one, or upgrade to a new bigtop version.
this upgrade should allow to test that spark3 is functional (dynamic execution, job submission in scala, python and R, notebooks)

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T307500 Airflow Hackathon (May 2022)
Resolved	Ahoelzl	T307505 Refine jobs should be scheduled by Airflow
Resolved	Antoine_Quhen	T356762 [Refine refactoring] Refine jobs should be scheduled by Airflow: implementation
Resolved	None	T291464 Upgrade analytics-hadoop to Spark 3 + scala 2.12
Duplicate	None	T291465 Analytics-test-hadoop Spark3 package upgrade
Duplicate	None	T291466 Analytics-hadoop Spark3 package upgrade (production)
Resolved	Ottomata	T291386 Upgrade Refinery Jobs to Spark 3
Open	None	T209453 Refine: Use Spark SQL instead of Hive JDBC
Open	Ottomata	T307040 Propagate field descriptions from event schemas to Hive event tables and into DataHub
Resolved	Antoine_Quhen	T388793 When doing ADD COLUMN to a struct under a map, Iceberg fails to SELECT it
Resolved	Ottomata	T255818 Refine drops $schema field values
Open	None	T259924 HiveExtensions.convertToSchema does not properly convert arrays of structs
Resolved	tchin	T366487 Event Platform schemas should not support type changes to structs as array element or map value types

Event Timeline

JAllemandou created this task.Sep 21 2021, 8:39 AM

• odimitrijevic added a project: Data-Engineering.Sep 23 2021, 4:29 PM

We decided to wait for bigtop package and can proceed with testing then.

Question on this decision: This seems to force the upgrade to spark3 to be part of the bigger upgrade to BigTop3, that is altogether a quite big beast (hadoop2->3, most noticeably). The upgrade to BigTop3 is a big project, the previous similar one being the move to Bigtop took about a year to Luca and myself. If we wish to make this happen for Spark to Happen, I 'd like us to start planning on this sooner rather than later.

In a recent discussion we decided to separate the spark upgrade from the bigtop upgrade

Milimetric reopened this task as Open.Oct 28 2021, 9:50 PM

• odimitrijevic moved this task from Incoming (new tickets) to Transform on the Data-Engineering board.Nov 1 2021, 4:48 PM

Ottomata added a subtask: T295072: Install spark3 in analytics clusters.Nov 12 2021, 9:51 PM

Ottomata removed a subtask: T295072: Install spark3 in analytics clusters.

• odimitrijevic removed a project: Analytics.Jan 12 2022, 12:32 AM

Ottomata closed this task as a duplicate of T295072: Install spark3 in analytics clusters.May 11 2022, 5:45 PM

Analytics-test-hadoop Spark3 package upgradeClosed, DuplicatePublicActions

Description

Related ObjectsSearch...

Event Timeline

Analytics-test-hadoop Spark3 package upgrade
Closed, DuplicatePublic
Actions

Related Objects
Search...