Page MenuHomePhabricator
Create Task
Maniphest T334095

[Airflow] Migrate Netflow Druid loading jobs
Closed, ResolvedPublic9 Estimated Story Points
Actions

Description

We have 6 systemd timers that are triggering Druid ingestion of netflow-related datasets.
We should migrate them to Airflow.

The systemd timers are:

  1. netflow druid hourly loading
  2. netflow druid daily loading
  3. netflow druid sanitization
  4. network_flows_internal druid hourly loading
  5. network_flows_internal druid daily loading
  6. network_flows_internal druid sanitization

We can group them into 2 DAG files:

  1. Netflow DAG file, containing:
    • Hourly DAG
    • Daily DAG
    • Sanitization DAG
  2. Nework_flows_internal DAG file, containing:
    • Hourly DAG
    • Daily DAG
    • Sanitization DAG

Details

SubjectRepoBranchLines +/-
::analytics::refinery::job::druid_load: remove remaining jobsoperations/puppetproduction+0 -84
::analytics::refinery::job::druid_load: absent remaining jobsoperations/puppetproduction+5 -1
analytics::refinery::job::druid_load: Remove remaining jobsoperations/puppetproduction+0 -84
analytics::refinery::job::druid_load: absent all jobsoperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

mforns created this task.Apr 5 2023, 2:29 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptApr 5 2023, 2:29 PM
mforns added a comment.Apr 5 2023, 2:30 PM

This is all tested and in code review for a final approval:
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/217

mforns moved this task from Next Up to In Review on the Data Pipelines (Sprint 11) board.Apr 5 2023, 2:30 PM
mforns set the point value for this task to 9.Apr 5 2023, 3:07 PM
JArguello-WMF moved this task from In Review to Ready to Deploy on the Data Pipelines (Sprint 11) board.Apr 5 2023, 4:14 PM
gerritbot added a comment.Apr 6 2023, 9:22 PM

Change 906660 had a related patch set uploaded (by Mforns; author: Mforns):

[operations/puppet@production] analytics::refinery::job::druid_load: absent all jobs

https://gerrit.wikimedia.org/r/906660

gerritbot added a project: Patch-For-Review.Apr 6 2023, 9:22 PM
gerritbot added a comment.Apr 6 2023, 9:30 PM

Change 906662 had a related patch set uploaded (by Mforns; author: Mforns):

[operations/puppet@production] analytics::refinery::job::druid_load: Remove remaining jobs

https://gerrit.wikimedia.org/r/906662

gerritbot added a comment.Apr 6 2023, 9:33 PM

Change 906660 abandoned by Mforns:

[operations/puppet@production] analytics::refinery::job::druid_load: absent all jobs

Reason:

Messed up creating this on top of an old change.

https://gerrit.wikimedia.org/r/906660

gerritbot added a comment.Apr 6 2023, 9:33 PM

Change 906662 abandoned by Mforns:

[operations/puppet@production] analytics::refinery::job::druid_load: Remove remaining jobs

Reason:

Messed up creating this on top of an old change.

https://gerrit.wikimedia.org/r/906662

gerritbot added a comment.Apr 6 2023, 9:37 PM

Change 906665 had a related patch set uploaded (by Mforns; author: Mforns):

[operations/puppet@production] ::analytics::refinery::job::druid_load: absent remaining jobs

https://gerrit.wikimedia.org/r/906665

gerritbot added a comment.Apr 6 2023, 9:47 PM

Change 906667 had a related patch set uploaded (by Mforns; author: Mforns):

[operations/puppet@production] ::analytics::refinery::job::druid_load: remove remaining jobs

https://gerrit.wikimedia.org/r/906667

gerritbot added a comment.Apr 11 2023, 12:52 PM

Change 906665 merged by Ottomata:

[operations/puppet@production] ::analytics::refinery::job::druid_load: absent remaining jobs

https://gerrit.wikimedia.org/r/906665

mforns moved this task from Ready to Deploy to Done on the Data Pipelines (Sprint 11) board.Apr 11 2023, 8:22 PM
JArguello-WMF closed this task as Resolved.Apr 13 2023, 1:23 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits