Spike: Evaluate datahub schema versioning support
Closed, ResolvedPublic2 Estimated Story Points
Actions

Assigned To

Authored By

	• EChetty
	May 5 2022, 4:08 PM

Description

Spike Goal

Determine what the user experience is for versioning schemas as data changes (Test on Hive, then to event platform)

Key Questions:

What happens with breaking Schema changes?

DataHub tracks schema versions in accordance with SemVer, autmatically creating change events. These are available from the Timeline API.

Can we highlight Backwards compatible changes?

DataHub automatically does this with minor SemVer version increments.

Can we highlight Backwards incompatible changes?

DataHub automatically does this with major SemVer version increments.

What happens when we Upgrade datahub?

On upgrading from 0.8.28 to 0.8.32 and then subsequently 0.8.34, we observed no breakage in the history of schemas, all versioning was kept.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		BTullis	T308052 Upgrade Datahub
		Resolved		Milimetric	T307716 Spike: Evaluate datahub schema versioning support

Event Timeline

• EChetty created this task.May 5 2022, 4:08 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 5 2022, 4:08 PM

• EChetty moved this task from Next Up to Backlog on the Data-Catalog board.May 5 2022, 4:08 PM

• EChetty moved this task from Backlog to Next Up on the Data-Catalog board.May 5 2022, 4:16 PM

• EChetty set the point value for this task to 2.

• EChetty moved this task from Next Up to MVP on the Data-Catalog board.May 10 2022, 5:42 PM

This looks to be available behind the scenes but just not surfaced in the UI yet? https://datahubproject.io/docs/dev-guides/timeline/

• EChetty moved this task from MVP to Next Up on the Data-Catalog board.May 10 2022, 6:02 PM

• EChetty moved this task from Next Up to MVP on the Data-Catalog board.

I will work on this first, using my hive database, milimetric, and reporting findings here.

Milimetric moved this task from Next Up to In Progress on the Data-Engineering-Kanban board.May 10 2022, 8:17 PM

Milimetric updated the task description. (Show Details)May 10 2022, 9:20 PM

Milimetric updated the task description. (Show Details)

Milimetric moved this task from In Progress to In Code Review on the Data-Engineering-Kanban board.May 10 2022, 9:22 PM

Milimetric updated the task description. (Show Details)May 10 2022, 9:27 PM

• EChetty moved this task from MVP to Next Up on the Data-Catalog board.May 12 2022, 12:58 PM

We need to upgrade to DataHub 0.8.34 before this will work. I've added that ticket as a parent of this one.

Milimetric moved this task from Next Up to In Review on the Data-Catalog board.May 13 2022, 7:42 PM

I have upgraded DataHub to version 0.8.34 and now the Blame View is accessible, which is the first way to visualize schema changes over time.

https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,milimetric.test_ingestion,PROD)/Schema?is_lineage_mode=false

• EChetty updated the task description. (Show Details)May 24 2022, 2:49 PM

• EChetty moved this task from In Review to Done on the Data-Catalog board.

Milimetric moved this task from In Code Review to Done on the Data-Engineering-Kanban board.May 25 2022, 8:29 PM

Milimetric updated the task description. (Show Details)Jun 1 2022, 6:51 PM