Upgrade to Spark 3.2 to support Spark lineage for Iceberg tables
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	tchin
	Nov 3 2024, 3:12 PM

Description

When running a Spark job with DataHub's Spark lineage connector on an Iceberg table, it fails with this error:

12:03:12.188 [spark-listener-group-shared] INFO io.openlineage.spark.agent.util.PlanUtils - apply method failed with
java.lang.NoSuchMethodError: org.apache.iceberg.spark.SparkSessionCatalog.icebergCatalog()Lorg/apache/iceberg/catalog/Catalog;
	at io.openlineage.spark3.agent.lifecycle.plan.catalog.IcebergHandler.getIcebergTable(IcebergHandler.java:171)
	at io.openlineage.spark3.agent.lifecycle.plan.catalog.IcebergHandler.getDatasetVersion(IcebergHandler.java:156)
	at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.getDatasetVersion(CatalogUtils3.java:100)
	at io.openlineage.spark3.agent.utils.DatasetVersionDatasetFacetUtils.extractVersionFromDataSourceV2Relation(DatasetVersionDatasetFacetUtils.java:46)

The first time the icebergCatalog() method appears in the source code appears to be in Iceberg v1.2 for Spark 3.2. It does not exist in Iceberg v1.2 for Spark 3.1.

If we want to support Spark lineage for Iceberg tables, the first (and hopefully only) step is to upgrade to Spark 3.2.

Related Objects
Search...

Status	Assigned	Task
Open	None	T369756 [EPIC] Datahub Improvements
Open	None	T377789 Data Platform Data Lineage
Resolved	tchin	T306896 Integrate Spark with DataHub with lineage (Data-Engineering)
Open	None	T333013 [Iceberg Migration] Apache Iceberg Migration
Open	None	T338057 Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0
Open	None	T378899 Upgrade to Spark 3.2 to support Spark lineage for Iceberg tables

Event Timeline

tchin created this task.Nov 3 2024, 3:12 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptNov 3 2024, 3:12 PM

Ottomata renamed this task from Upgrade to Spark 3.2 to support Spark lineage to Upgrade to Spark 3.2 to support Spark lineage for Iceberg tables.Nov 4 2024, 5:42 PM

Ottomata added a parent task: T338057: Upgrade Spark to a version with long term Iceberg support, and with fixes to support Dumps 2.0.

tchin removed tchin as the assignee of this task.Nov 5 2024, 7:49 AM

Ahoelzl moved this task from Next Up to Blocked/Paused on the Data-Engineering (Q2 2024 October 1st - December 31th) board.Dec 18 2024, 1:54 PM

Ahoelzl edited projects, added Data-Engineering (Q3 2025 January 1st - March 31th); removed Data-Engineering (Q2 2024 October 1st - December 31th).Jan 18 2025, 12:31 AM

Ahoelzl moved this task from Next Up to Blocked/Paused on the Data-Engineering (Q3 2025 January 1st - March 31th) board.

Ahoelzl edited projects, added Data-Engineering; removed Data-Engineering (Q3 2025 January 1st - March 31th).Jan 24 2025, 6:36 PM