Root Epic for accomplishing comprehensive data lineage for the Data Platform.
Given the complexity and heterogeneity of systems of the Data Platform, the work to achieve full data lineage should be further broken down into milestones targeting meaningful areas of lineage coverage.
Milestones
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T369756 [EPIC] Datahub Improvements | |||
| Open | None | T377789 Data Platform Data Lineage | |||
| Resolved | tchin | T306896 Integrate Spark with DataHub with lineage (Data-Engineering) | |||
| Resolved | tchin | T372899 Ingest a test hive database into datahub | |||
| Open | None | T378899 Upgrade to Spark 3.2 to support Spark lineage for Iceberg tables | |||
| Resolved | Ahoelzl | T369758 [SPIKE] Define process to build out lineage in DataHub | |||
| Duplicate | None | T386724 Integrate Spark with DataHub with lineage (non Data-Engineering Airflow instances) | |||
| Open | mforns | T386862 Enable Spark data lineage for all Airflow instances |