Page MenuHomePhabricator
Create Task
Maniphest T305842

Migrate the referrer job
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Goal:
Migrate the referrer job to Airflow:

Job Details:

InputProcessingOutput
Hive TableHiveHive & Archive

Success Criteria:

  • Have the 1 Daily Job Migrated (SLA 6 Hours)
Gotchas
  • This job includes archiving of results. Maybe we need to adapt the existing Airflow custom ArchiveOperator to match this job's format.

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT307500 Airflow Hackathon (May 2022)
 ResolvedJEbe-WMFT305842 Migrate the referrer job

Event Timeline

• EChetty created this task.Apr 11 2022, 1:16 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 11 2022, 1:16 PM
mforns renamed this task from Migrate the Referrer jobs to Airflow to Migrate the Referrer job.May 4 2022, 1:22 PM
mforns updated the task description. (Show Details)
mforns added a parent task: T307500: Airflow Hackathon (May 2022).
mforns updated the task description. (Show Details)
mforns added a project: Data-Engineering.
mforns renamed this task from Migrate the Referrer job to Migrate the referrer job.May 4 2022, 1:56 PM
mforns moved this task from Incoming (new tickets) to Transform on the Data-Engineering board.May 11 2022, 6:45 PM
Antoine_Quhen claimed this task.May 24 2022, 4:07 PM
Antoine_Quhen moved this task from Discussed (Radar) to Estimated on the Data Pipelines board.
Antoine_Quhen added a project: Data-Engineering-Kanban.May 24 2022, 4:13 PM
Antoine_Quhen moved this task from Next Up to In Progress on the Data-Engineering-Kanban board.
JArguello-WMF moved this task from Estimated to Backlog on the Data Pipelines board.Jun 29 2022, 2:45 PM
JArguello-WMF edited projects, added Data-Engineering-Planning; removed Data-Engineering.Jun 30 2022, 3:15 PM
JArguello-WMF edited projects, added Data-Engineering-Planning (Sprint 01); removed Data-Engineering-Planning.Jun 30 2022, 3:33 PM
JArguello-WMF removed projects: Data-Engineering-Planning (Sprint 01), Data-Engineering-Kanban.Jun 30 2022, 3:43 PM
Antoine_Quhen edited projects, added Data Pipelines (sprint 10); removed Data Pipelines.Feb 21 2023, 5:31 PM
Antoine_Quhen edited projects, added Data Pipelines (Sprint 11); removed Data Pipelines (sprint 10).Feb 21 2023, 5:35 PM
JArguello-WMF moved this task from Next Up to Next Up on the Data Pipelines (Sprint 11) board.Feb 21 2023, 5:45 PM
JArguello-WMF set the point value for this task to 3.
Antoine_Quhen moved this task from Next Up to In Progress on the Data Pipelines (Sprint 11) board.Mar 7 2023, 5:04 PM
JArguello-WMF edited projects, added Data Pipelines (sprint 10); removed Data Pipelines (Sprint 11).Mar 13 2023, 1:57 PM
JArguello-WMF moved this task from Ready to In Progress on the Data Pipelines (sprint 10) board.
JArguello-WMF removed Antoine_Quhen as the assignee of this task.Mar 15 2023, 5:05 PM
JArguello-WMF moved this task from In Progress to Next Up on the Data Pipelines (sprint 10) board.
JArguello-WMF added a subscriber: Antoine_Quhen.
JEbe-WMF claimed this task.Mar 16 2023, 3:01 PM
JEbe-WMF moved this task from Next Up to In Progress on the Data Pipelines (sprint 10) board.
gerritbot added a comment.Mar 22 2023, 12:14 PM

Change 902068 had a related patch set uploaded (by Jennifer Ebe; author: Jennifer Ebe):

[analytics/refinery@master] T305842-Migrate-The-Referrer-Job-Daily-hql Create archive and referrer hql for referrer daily airflow job

https://gerrit.wikimedia.org/r/902068

gerritbot added a project: Patch-For-Review.Mar 22 2023, 12:14 PM
JArguello-WMF moved this task from In Progress to In Review on the Data Pipelines (sprint 10) board.Mar 22 2023, 5:05 PM
gerritbot added a comment.Mar 28 2023, 9:24 AM

Change 902068 merged by Joal:

[analytics/refinery@master] T305842-Migrate-The-Referrer-Job-Daily-hql

https://gerrit.wikimedia.org/r/902068

Maintenance_bot removed a project: Patch-For-Review.Mar 28 2023, 9:30 AM
JArguello-WMF moved this task from In Review to Ready to Deploy on the Data Pipelines (sprint 10) board.Mar 28 2023, 4:04 PM
JArguello-WMF moved this task from Ready to Deploy to Done on the Data Pipelines (sprint 10) board.Mar 30 2023, 4:03 PM
JArguello-WMF closed this task as Resolved.Mar 31 2023, 1:54 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits