Page MenuHomePhabricator
Create Task
Maniphest T334105

[Airflow] Migrate virtualpageviews Druid loading Oozie jobs
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

We have 2 virtualpageviews druid loading jobs in Oozie:

  1. daily
  2. monthly

We can group them both in a single DAG file with 2 DAGs.

Details

SubjectRepoBranchLines +/-
Adapt virtualpageview druid scripts to sparkanalytics/refinerymaster+180 -0
Customize query in gerrit
TitleReferenceAuthorSource BranchDest Branch
Add druid loading jobs for virtualpageview_hourlyrepos/data-engineering/airflow-dags!367milimetricmigrate-druid-load-virtualpageviewmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

mforns created this task.Apr 5 2023, 3:24 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptApr 5 2023, 3:24 PM
mforns set the point value for this task to 3.Apr 5 2023, 3:24 PM
Milimetric claimed this task.Apr 20 2023, 8:15 PM
Milimetric moved this task from Next Up to In Progress on the Data Pipelines (Sprint 11) board.
JArguello-WMF edited projects, added Data Pipelines (Sprint 12); removed Data Pipelines (Sprint 11).Apr 24 2023, 1:17 PM
JArguello-WMF moved this task from Next Up to In Progress on the Data Pipelines (Sprint 12) board.
gerritbot added a comment.Apr 26 2023, 7:17 PM

Change 912360 had a related patch set uploaded (by Milimetric; author: Milimetric):

[analytics/refinery@master] Adapt virtualpageview druid scripts to spark

https://gerrit.wikimedia.org/r/912360

gerritbot added a project: Patch-For-Review.Apr 26 2023, 7:17 PM
gerritbot added a comment.Apr 26 2023, 7:58 PM

milimetric opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/367

Draft: Add druid loading jobs for virtualpageview_hourly

Milimetric moved this task from In Progress to In Review on the Data Pipelines (Sprint 12) board.May 3 2023, 2:06 PM
Milimetric set Final Story Points to 5.
gerritbot added a comment.May 4 2023, 1:41 PM

milimetric merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/367

Add druid loading jobs for virtualpageview_hourly

mforns moved this task from In Review to Ready to Deploy on the Data Pipelines (Sprint 12) board.May 5 2023, 2:55 PM
gerritbot added a comment.May 10 2023, 3:43 PM

Change 912360 merged by Milimetric:

[analytics/refinery@master] Adapt virtualpageview druid scripts to spark

https://gerrit.wikimedia.org/r/912360

Milimetric moved this task from Ready to Deploy to Done on the Data Pipelines (Sprint 12) board.May 10 2023, 9:32 PM
JArguello-WMF closed this task as Resolved.May 12 2023, 3:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits