Page MenuHomePhabricator
Create Task
Maniphest T334096

[Airflow] Migrate unique devices Druid loading jobs
Closed, ResolvedPublic9 Estimated Story Points
Actions

Description

We have 6 Oozie jobs that load unique devices data to Druid.

  1. unique devices per domain daily
  2. unique devices per domain daily aggregated monthly
  3. unique devices per domain monthly
  4. unique devices per project family daily
  5. unique devices per project family daily aggregated monthly
  6. unique devices per project family monthly

We can group them into 2 DAG files:

  1. unique devices per domain DAG file, with:
    • daily DAG
    • daily aggregated monthly DAG
    • monthly DAG
  2. unique devices per project family DAG file, with:
    • daily DAG
    • daily aggregated monthly DAG
    • monthly DAG

Details

SubjectRepoBranchLines +/-
Migrate unique devices druid loading queries to Airflow/SparkSQLanalytics/refinerymaster+321 -0
Fix HiveToDruid to allow for non-partitioned source tables.analytics/refinery/sourcemaster+10 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

mforns created this task.Apr 5 2023, 2:40 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptApr 5 2023, 2:40 PM
mforns added a comment.EditedApr 5 2023, 2:52 PM

This is currently in review for DAG file 1) and in progress for DAG file 2:
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/350

mforns moved this task from Next Up to In Progress on the Data Pipelines (Sprint 11) board.Apr 5 2023, 2:56 PM
mforns set the point value for this task to 9.Apr 5 2023, 3:08 PM
JArguello-WMF closed subtask T334098: [Airflow] Improve VariableProperties as Resolved.Apr 13 2023, 1:23 PM
gerritbot added a comment.Apr 19 2023, 8:39 PM

Change 910092 had a related patch set uploaded (by Mforns; author: Mforns):

[analytics/refinery@master] Migrate unique devices druid loading queries to Airflow/SparkSQL

https://gerrit.wikimedia.org/r/910092

gerritbot added a project: Patch-For-Review.Apr 19 2023, 8:39 PM
gerritbot added a comment.Apr 19 2023, 8:44 PM

Change 910094 had a related patch set uploaded (by Mforns; author: Mforns):

[analytics/refinery/source@master] Fix HiveToDruid to allow for non-partitioned source tables.

https://gerrit.wikimedia.org/r/910094

mforns moved this task from In Progress to In Review on the Data Pipelines (Sprint 11) board.Apr 19 2023, 8:49 PM

I've tested the DAGs in the dev instance and all seems to work! This is in code review :-)

JArguello-WMF edited projects, added Data Pipelines (Sprint 12); removed Data Pipelines (Sprint 11).Apr 24 2023, 1:18 PM
JArguello-WMF moved this task from Next Up to In Review on the Data Pipelines (Sprint 12) board.
JArguello-WMF moved this task from In Review to Ready to Deploy on the Data Pipelines (Sprint 12) board.Apr 24 2023, 4:05 PM
JArguello-WMF moved this task from Ready to Deploy to In Review on the Data Pipelines (Sprint 12) board.
mforns moved this task from In Review to Ready to Deploy on the Data Pipelines (Sprint 12) board.Apr 26 2023, 4:00 PM
Milimetric moved this task from Ready to Deploy to In Review on the Data Pipelines (Sprint 12) board.Apr 27 2023, 4:04 PM
mforns moved this task from In Review to Ready to Deploy on the Data Pipelines (Sprint 12) board.May 3 2023, 4:07 PM
gerritbot added a comment.May 3 2023, 5:47 PM

Change 910094 merged by jenkins-bot:

[analytics/refinery/source@master] Fix HiveToDruid to allow for non-partitioned source tables.

https://gerrit.wikimedia.org/r/910094

gerritbot added a comment.May 3 2023, 6:13 PM

Change 910092 merged by Mforns:

[analytics/refinery@master] Migrate unique devices druid loading queries to Airflow/SparkSQL

https://gerrit.wikimedia.org/r/910092

mforns moved this task from Ready to Deploy to Done on the Data Pipelines (Sprint 12) board.May 8 2023, 4:03 PM
JArguello-WMF closed this task as Resolved.May 12 2023, 3:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits