[L] Orchestrate image suggestions tasks with Apache Airflow
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	mfossati
	Feb 23 2022, 7:10 PM

Description

User story

As a developer, I need to schedule and orchestrate the various data gathering and storing tasks for image suggestions, so that image suggestions data is available to users.

Tasks

write an Airflow job following PETS's guidelines
open a draft merge request to https://gitlab.wikimedia.org/repos/generated-data-platform/datapipelines/: https://gitlab.wikimedia.org/repos/generated-data-platform/datapipelines/-/merge_requests/51
fix eventual CI faillures
create a phab task as per PET's instructions, point 1: https://phabricator.wikimedia.org/T307356
~~agree with PET on a weekly schedule~~ set a weekly value for schedule_interval in the DAG script

Related Objects
Search...

Status	Assigned	Task
Resolved	CBogen	T299781 [EPIC] Image suggestions backend
Resolved	mfossati	T296814 [EPIC] Article-level image suggestions data pipeline
Resolved	mfossati	T302434 [L] Orchestrate image suggestions tasks with Apache Airflow

Event Timeline

mfossati created this task.Feb 23 2022, 7:10 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 23 2022, 7:10 PM

mfossati mentioned this in T296814: [EPIC] Article-level image suggestions data pipeline.Feb 23 2022, 7:14 PM

mfossati added a parent task: T296814: [EPIC] Article-level image suggestions data pipeline.

CBogen moved this task from Incoming to Ready for Estimation on the Structured-Data-Backlog (Current Work) board.Feb 23 2022, 7:45 PM

CBogen renamed this task from Orchestrate image suggestions tasks with Apache Airflow to [L] Orchestrate image suggestions tasks with Apache Airflow.Mar 9 2022, 5:21 PM

CBogen moved this task from Ready for Estimation to Ready for Development on the Structured-Data-Backlog (Current Work) board.Mar 9 2022, 5:42 PM

mfossati claimed this task.Mar 14 2022, 5:03 PM

mfossati changed the task status from Open to In Progress.Apr 4 2022, 8:12 AM

mfossati moved this task from Ready for Development to Doing on the Structured-Data-Backlog (Current Work) board.

mfossati updated the task description. (Show Details)Apr 5 2022, 2:59 PM

mfossati updated the task description. (Show Details)Apr 6 2022, 9:10 AM

Blocked by:

Currently open tasks are the final packaging steps before we wrap the whole data pipeline up.
Moving to blocked.

Now that we have access to the Airflow instance, we can keep adding the tasks to the Airflow DAG, and launch & monitor their execution.
Moving back to doing.

mfossati updated the task description. (Show Details)May 2 2022, 11:47 AM

mfossati updated the task description. (Show Details)May 2 2022, 1:49 PM