Maniphest T308050

Airflow Job for Ingesting Kafka Metadata into DataHub
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• EChetty
	May 10 2022, 5:49 PM

Tags

Referenced Files

None

Subscribers

Description

Primary Task

Setup airflow jobs to schedule the ingestion and publishing of metadata to Datahub from Kafka

Key tasks:

Profile the performance of the update schedule.
Define and Document Airflow DAGs for metadata publishing.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Milimetric	T307714 Custom Metadata ingestion
		Resolved		Milimetric	T308050 Airflow Job for Ingesting Kafka Metadata into DataHub

Event Timeline

• EChetty created this task.May 10 2022, 5:49 PM

• EChetty moved this task from Backlog to MVP on the Data-Catalog board.

• EChetty renamed this task from Airflow: kafka to Airflow Job for Ingesting Kafka Metadata into DataHub.May 12 2022, 12:49 PM

• EChetty updated the task description. (Show Details)

• EChetty added a project: Data Pipelines.May 12 2022, 1:07 PM

• EChetty assigned this task to Milimetric.May 24 2022, 2:54 PM

• EChetty moved this task from MVP to In Progress on the Data-Catalog board.

• EChetty moved this task from Backlog to Discussed (Radar) on the Data Pipelines board.Jun 6 2022, 3:40 PM

• EChetty moved this task from Discussed (Radar) to Estimated on the Data Pipelines board.

• EChetty moved this task from In Progress to In Review on the Data-Catalog board.Jun 7 2022, 2:52 PM

mforns moved this task from Estimated to In Review on the Data Pipelines board.Jun 13 2022, 3:41 PM

Milimetric moved this task from In Review to Done on the Data-Catalog board.Jun 28 2022, 5:25 PM

JArguello-WMF moved this task from In Review to Sprint 00 on the Data Pipelines board.Jul 28 2022, 2:13 PM

JArguello-WMF edited projects, added Data Pipelines (Sprint 00); removed Data Pipelines.

• EChetty removed a project: Data Pipelines (Sprint 00).Aug 15 2022, 9:01 AM

Milimetric closed this task as Resolved.Aug 23 2022, 5:18 PM