Page MenuHomePhabricator
Create Task
Maniphest T311825

[M] Create the section-level image suggestions Airflow DAG
Closed, ResolvedPublic
Actions

Description

Once T311814: [EPIC] Section-level image suggestions data pipeline's Spark code is ready, create the Airflow DAG.

NOTE: Data Platform is migrating the DAGs to a new repository with different Airflow operator wrappers, see T311417: Migration of Image Suggestion Job to DE Data Pipeline. This DAG should comply with the new conventions.

Related Objects
Search...

Event Timeline

mfossati created this task.Jul 1 2022, 10:51 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 1 2022, 10:51 AM
mfossati added a parent task: T311814: [EPIC] Section-level image suggestions data pipeline.Jul 1 2022, 11:20 AM
CBogen moved this task from Triage to Section Topics on the Structured-Data-Backlog board.Jul 18 2022, 7:18 PM
CBogen moved this task from Section Topics to Section Level Image Suggestions on the Structured-Data-Backlog board.
CBogen edited projects, added Structured-Data-Backlog (Current Work); removed Structured-Data-Backlog.Dec 12 2022, 4:56 PM
CBogen moved this task from Incoming to Ready for Estimation on the Structured-Data-Backlog (Current Work) board.
CBogen renamed this task from Create the section-level image suggestions Airflow DAG to [M] Create the section-level image suggestions Airflow DAG.Dec 14 2022, 5:45 PM
CBogen moved this task from Ready for Estimation to Ready for Development on the Structured-Data-Backlog (Current Work) board.Dec 14 2022, 6:00 PM
mfossati mentioned this in T324588: Schedule the section-image-recommendations based on section-alignment code.Dec 14 2022, 6:07 PM
mfossati moved this task from Ready for Development to Blocked on the Structured-Data-Backlog (Current Work) board.Dec 14 2022, 7:04 PM

Moving to blocked since this ticket shouldn’t be worked on until the whole data processing logic is ready.
Blocked on:

Cparle added subscribers: xcollazo, matthiasmullie, Cparle.Mar 13 2023, 4:31 PM

I propose that we just add this to the image-suggestions DAG rather than it having its own DAG. @mfossati @xcollazo @matthiasmullie what do you think?

mfossati added a comment.Mar 15 2023, 12:58 PM
In T311825#8688274, @Cparle wrote:

I propose that we just add this to the image-suggestions DAG rather than it having its own DAG. @mfossati @xcollazo @matthiasmullie what do you think?

Sounds good to me.

xcollazo added a comment.Mar 15 2023, 2:51 PM
In T311825#8697862, @mfossati wrote:
In T311825#8688274, @Cparle wrote:

I propose that we just add this to the image-suggestions DAG rather than it having its own DAG. @mfossati @xcollazo @matthiasmullie what do you think?

Sounds good to me.

👍

xcollazo closed this task as Resolved.Mar 31 2023, 2:23 AM
xcollazo claimed this task.

Done via T330667, specifically https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/327.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL