Page MenuHomePhabricator
Create Task
Maniphest T307356

Image suggestions data pipeline onboarding request
Closed, ResolvedPublic
Actions

Description

Draft merge request at https://gitlab.wikimedia.org/repos/generated-data-platform/datapipelines/-/merge_requests/51. marked as ready to be merged

From https://www.mediawiki.org/wiki/Platform_Engineering_Team/Data_Value_Stream/Data_Pipeline_Onboarding/#Onboarding:

Related Objects
Search...

Event Timeline

mfossati created this task.May 2 2022, 1:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 2 2022, 1:46 PM
mfossati mentioned this in T302434: [L] Orchestrate image suggestions tasks with Apache Airflow.May 2 2022, 1:49 PM
mfossati added a subtask: T307362: Memory errors break the expected output of some Airflow tasks.May 2 2022, 3:18 PM
mfossati updated the task description. (Show Details)
mfossati added a subtask: T307371: Data pipeline weekly schedule is on hold.May 2 2022, 4:09 PM
mfossati updated the task description. (Show Details)
mfossati changed the status of subtask T307371: Data pipeline weekly schedule is on hold from Open to In Progress.May 5 2022, 8:51 AM
mfossati changed the status of subtask T307362: Memory errors break the expected output of some Airflow tasks from Open to In Progress.
mfossati added a subtask: T307899: Data pipeline grooming.May 9 2022, 4:00 PM
mfossati changed the status of subtask T307899: Data pipeline grooming from Open to In Progress.May 9 2022, 4:33 PM
mfossati closed subtask T307362: Memory errors break the expected output of some Airflow tasks as Resolved.May 13 2022, 3:26 PM
mfossati closed subtask T307899: Data pipeline grooming as Resolved.May 16 2022, 12:30 PM
CBogen assigned this task to mfossati.May 16 2022, 4:36 PM
CBogen edited projects, added Structured-Data-Backlog (Current Work); removed Structured-Data-Backlog.
CBogen moved this task from Incoming to Doing on the Structured-Data-Backlog (Current Work) board.
mfossati changed the task status from Open to In Progress.May 17 2022, 10:48 AM
mfossati closed subtask T307371: Data pipeline weekly schedule is on hold as Resolved.
mfossati added a parent task: T296814: [EPIC] Article-level image suggestions data pipeline.May 17 2022, 10:50 AM
mfossati updated the task description. (Show Details)
Cparle subscribed.May 23 2022, 4:24 PM

Waiting for the patch to be merged before closing this

• lbowmaker moved this task from Backlog to Work in Progress ⚙️ on the Generated Data Platform board.Jun 7 2022, 11:45 AM
mfossati updated the task description. (Show Details)Jun 8 2022, 10:32 AM
mfossati moved this task from Doing to Code Review on the Structured-Data-Backlog (Current Work) board.Jun 8 2022, 10:38 AM

Note that code review comes from Data Platform.

CBogen moved this task from Backlog to Data Pipeline on the Image-Suggestions board.Jul 7 2022, 6:37 PM
CBogen subscribed.Jul 18 2022, 7:25 PM

@Cparle @mfossati can this ticket be closed?

mfossati added a comment.Jul 19 2022, 1:28 PM

@CBogen this is still open since code review & merge hasn't happened yet

mfossati closed this task as Resolved.Aug 4 2022, 3:35 PM

Closing this: Data Platform has onboarded the pipeline.

NOTE: while the Airflow DAG was reviewed by Data Platform, the pyspark components were reviewed @Cparle and me.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits