Refine to Airflow Migration: User Story
Closed, DuplicatePublic
Actions

Assigned To

None

Authored By

	• EChetty
	Nov 26 2021, 1:44 PM

Description

User Story

As a data engineer, I want to begin consolidating our refine jobs into Airflow, so that I can get faster at deploying, maintaining and optimising our Data Pipelines

Success Criteria

Deploy our POC jobs into our production instance of airflow, complete with monitoring and testing capabilities.
Have all our Refine Scheduler jobs moved into our airflow instance.
Deprecate Refine - ensuring that its only available through an airflow operator.

Open questions / remarks

Is our POC the right Job to start with?
Do we have all the required operators?
Who needs to validate that the pipeline is working as intended?

Event Timeline

• EChetty created this task.Nov 26 2021, 1:44 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 26 2021, 1:44 PM

• EChetty renamed this task from Refine to Airflow Migration to Refine to Airflow Migration: User Story.Nov 26 2021, 1:52 PM

mforns added a project: Epic.Nov 26 2021, 2:27 PM

• EChetty moved this task from Backlog to Epics on the Data Pipelines board.Jan 25 2022, 11:32 AM

We should probably move ProduceCanaryEvents to Airflow first. It will be similar technically, but easier and safer to implement and migrate.

JArguello-WMF added a project: Data Engineering and Event Platform Team.Jun 30 2023, 5:31 PM

JArguello-WMF moved this task from Data Eng Backlog to Parent Tasks/Epics on the Data Engineering and Event Platform Team board.Jun 30 2023, 5:38 PM

Ottomata closed this task as a duplicate of T307505: Refine jobs should be scheduled by Airflow.Jul 6 2023, 1:57 PM

Refine to Airflow Migration: User StoryClosed, DuplicatePublicActions