Page MenuHomePhabricator
Create Task
Maniphest T334106

[Airflow] Migrate webrequest sampled 128 druid loading jobs
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

We have 2 Oozie jobs that load sampled webrequest data into Druid
An hourly one and a daily one.
We can group them both in the same DAG file.

Details

SubjectRepoBranchLines +/-
Fix webrequest sampled 128 druid loading queriesanalytics/refinerymaster+2 -2
Migrate queries for webrequest_sampled_128 to /hql (Airflow/Spark3)analytics/refinerymaster+268 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

mforns created this task.Apr 5 2023, 3:27 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptApr 5 2023, 3:27 PM
mforns set the point value for this task to 3.Apr 5 2023, 3:28 PM
mforns claimed this task.Apr 20 2023, 1:20 PM
mforns moved this task from Next Up to In Progress on the Data Pipelines (Sprint 11) board.
JArguello-WMF edited projects, added Data Pipelines (Sprint 12); removed Data Pipelines (Sprint 11).Apr 24 2023, 1:17 PM
JArguello-WMF moved this task from Next Up to In Progress on the Data Pipelines (Sprint 12) board.
gerritbot added a comment.Apr 25 2023, 3:46 PM

Change 911890 had a related patch set uploaded (by Mforns; author: Mforns):

[analytics/refinery@master] Migrate queries for webrequest_sampled_128 to /hql (Airflow/Spark3)

https://gerrit.wikimedia.org/r/911890

gerritbot added a project: Patch-For-Review.Apr 25 2023, 3:46 PM
mforns added a comment.Apr 25 2023, 3:47 PM

And here's the airflow-dags MR:
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/364

mforns moved this task from In Progress to In Review on the Data Pipelines (Sprint 12) board.Apr 25 2023, 3:47 PM
Milimetric moved this task from In Review to Ready to Deploy on the Data Pipelines (Sprint 12) board.Apr 27 2023, 4:04 PM
Milimetric moved this task from Ready to Deploy to In Review on the Data Pipelines (Sprint 12) board.
mforns moved this task from In Review to Ready to Deploy on the Data Pipelines (Sprint 12) board.May 3 2023, 4:07 PM
gerritbot added a comment.May 3 2023, 5:35 PM

Change 911890 merged by Milimetric:

[analytics/refinery@master] Migrate queries for webrequest_sampled_128 to /hql (Airflow/Spark3)

https://gerrit.wikimedia.org/r/911890

gerritbot added a comment.May 5 2023, 5:50 PM

Change 916537 had a related patch set uploaded (by Mforns; author: Mforns):

[analytics/refinery@master] Fix webrequest sampled 128 druid loading queries

https://gerrit.wikimedia.org/r/916537

gerritbot added a comment.May 5 2023, 5:58 PM

Change 916537 merged by Milimetric:

[analytics/refinery@master] Fix webrequest sampled 128 druid loading queries

https://gerrit.wikimedia.org/r/916537

mforns moved this task from Ready to Deploy to Done on the Data Pipelines (Sprint 12) board.May 8 2023, 4:03 PM
JArguello-WMF closed this task as Resolved.May 12 2023, 3:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits