User Story

As a Data Engineer I need to support the migration of Product Analytics owned data pipelines to Airflow

Why?

Airflow is the chosen system by Data Engineering for scheduled data pipeline execution, management and support. As such, the DE team will encourage other teams to migrate their jobs to the platform so that we can have a consistent and well supported experience for data producers.

Success Criteria

• 1 simple Hive to Hive based data pipeline is migrated that instruments lineage information (more if implementing teams are able). Following simple steps drafted here
• Product Analytics instance of Airflow is setup
• Airflow deployment via Git repository artifacts is possible