Page MenuHomePhabricator
Create Task
Maniphest T334104

[Airflow] Migrate pageview-related Druid loading Oozie jobs
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

We have 3 Oozie jobs that load pageview data to Druid:

  1. Pageview druid hourly
  2. Pageview druid daily
  3. Pageview druid monthly

We can group them in 1 DAG file with:

  • hurly DAG
  • daily DAG
  • monthly DAG

Details

SubjectRepoBranchLines +/-
Migrate pageview druid load hql queries to Airflowanalytics/refinerymaster+515 -0
Customize query in gerrit
TitleReferenceAuthorSource BranchDest Branch
Migrate pageview druid loading jobs to airflowrepos/data-engineering/airflow-dags!365ebysansT334104_pageview_druidmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

mforns created this task.Apr 5 2023, 3:22 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptApr 5 2023, 3:22 PM
mforns set the point value for this task to 5.Apr 5 2023, 3:22 PM
mforns added a comment.Apr 5 2023, 4:23 PM

We should add the referer information to the Druid datasources as requested in https://phabricator.wikimedia.org/T331028 !

mforns mentioned this in T331028: NEW FEATURE REQUEST: <Update webrequest derived tables to use the new column for referer data>.Apr 5 2023, 4:24 PM
Snwachukwu claimed this task.Apr 13 2023, 2:06 PM
Snwachukwu moved this task from Next Up to In Progress on the Data Pipelines (Sprint 11) board.Apr 13 2023, 2:43 PM
gerritbot added a comment.Apr 20 2023, 3:30 PM

Change 910520 had a related patch set uploaded (by Snwachukwu; author: Snwachukwu):

[analytics/refinery@master] Migrate pageview druid load hql queries to Airflow

https://gerrit.wikimedia.org/r/910520

gerritbot added a project: Patch-For-Review.Apr 20 2023, 3:30 PM
JArguello-WMF moved this task from In Progress to In Review on the Data Pipelines (Sprint 11) board.Apr 20 2023, 4:04 PM
gerritbot added a comment.Apr 24 2023, 8:21 AM

ebysans opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/365

Migrate pageview druid loading jobs to airflow

JArguello-WMF edited projects, added Data Pipelines (Sprint 12); removed Data Pipelines (Sprint 11).Apr 24 2023, 1:18 PM
JArguello-WMF moved this task from Next Up to In Review on the Data Pipelines (Sprint 12) board.
JArguello-WMF moved this task from In Review to In Progress on the Data Pipelines (Sprint 12) board.Apr 26 2023, 4:07 PM
JArguello-WMF moved this task from In Progress to In Review on the Data Pipelines (Sprint 12) board.May 1 2023, 4:11 PM
gerritbot added a comment.May 9 2023, 5:59 PM

milimetric merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/365

Migrate pageview druid loading jobs to airflow

gerritbot added a comment.May 10 2023, 3:00 PM

Change 910520 merged by Snwachukwu:

[analytics/refinery@master] Migrate pageview druid load hql queries to Airflow

https://gerrit.wikimedia.org/r/910520

Snwachukwu moved this task from In Review to Ready to Deploy on the Data Pipelines (Sprint 12) board.May 10 2023, 3:48 PM
Milimetric moved this task from Ready to Deploy to Done on the Data Pipelines (Sprint 12) board.May 10 2023, 9:32 PM
JArguello-WMF closed this task as Resolved.May 12 2023, 3:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits