Maniphest T291466

Analytics-hadoop Spark3 package upgrade (production)
Closed, DuplicatePublic
Actions

Assigned To

None

Authored By

	JAllemandou
	Sep 21 2021, 8:40 AM

Project Tags

Referenced Files

None

Subscribers

Description

Once tested on the test cluster, let's upgrade the production one.
This task should be synchronized with the release of T291386.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		None	T307500 Airflow Hackathon (May 2022)
		Open		None	T307505 Refine jobs should be scheduled by Airflow
		Resolved		Antoine_Quhen	T356762 [Refine refactoring] Refine jobs should be scheduled by Airflow: implementation
		Resolved		None	T291464 Upgrade analytics-hadoop to Spark 3 + scala 2.12
		Duplicate		None	T291465 Analytics-test-hadoop Spark3 package upgrade
		Duplicate		None	T291466 Analytics-hadoop Spark3 package upgrade (production)
		Resolved		Ottomata	T291386 Upgrade Refinery Jobs to Spark 3
		Open		None	T209453 Refine: Use Spark SQL instead of Hive JDBC
		Declined		Ottomata	T307040 Propagate field descriptions from event schemas to Hive event tables and into DataHub
		Resolved		Antoine_Quhen	T388793 When doing ADD COLUMN to a struct under a map, Iceberg fails to SELECT it
		Resolved		Ottomata	T255818 Refine drops $schema field values
		Open		None	T259924 HiveExtensions.convertToSchema does not properly convert arrays of structs
		Resolved		tchin	T366487 Event Platform schemas should not support type changes to structs as array element or map value types

Event Timeline

JAllemandou created this task.Sep 21 2021, 8:40 AM

JAllemandou added a parent task: T291464: Upgrade analytics-hadoop to Spark 3 + scala 2.12.

JAllemandou added a subtask: T291386: Upgrade Refinery Jobs to Spark 3.

• odimitrijevic added a project: Data-Engineering.Sep 23 2021, 4:29 PM

• odimitrijevic triaged this task as Medium priority.Sep 27 2021, 4:14 PM

• odimitrijevic moved this task from Incoming to Smart Tools for Better Data on the Analytics board.

• odimitrijevic moved this task from Incoming (new tickets) to Transform on the Data-Engineering board.Nov 1 2021, 4:49 PM

• odimitrijevic removed a project: Analytics.Jan 12 2022, 12:32 AM

Ottomata closed this task as a duplicate of T295072: Install spark3 in analytics clusters.May 11 2022, 5:44 PM

Ottomata closed subtask T291386: Upgrade Refinery Jobs to Spark 3 as Resolved.Apr 28 2025, 12:53 PM