[Airflow] Migrate Druid loading Oozie jobs - Parent task
Closed, ResolvedPublic0 Estimated Story Points
Actions

Assigned To

Authored By

	JArguello-WMF
	Dec 5 2022, 4:54 PM

Description

Migrate all Oozie Druid-loading jobs to Airflow.
This task assumes we already have at least 1 Druid-loading job working in Airflow, that can be used as a reference.
The workload per Oozie job to migrate should be the following:

Write the DAG (copy-paste of the existing one, and modify according to the specific dataset and its data sources) and its unit tests.
Test the DAG in a dev-instance.
Code review.
Deployment, kill the old Oozie job, enable the new one.
Test that all is working in prod.

Details

	Subject	Repo	Branch	Lines +/-
	modules::profile::manifests::airflow.pp: add plugins_folder path	operations/puppet	production	+2 -0
	Support snapshot partitioning in HiveToDruid and DataFrameToDruid	analytics/refinery/source	master	+70 -216

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	JArguello-WMF	T324485 [Airflow] Migrate Druid loading Oozie jobs - Parent task
Resolved	mforns	T334095 [Airflow] Migrate Netflow Druid loading jobs
Resolved	mforns	T334096 [Airflow] Migrate unique devices Druid loading jobs
Resolved	mforns	T334098 [Airflow] Improve VariableProperties
Resolved	Antoine_Quhen	T334101 [Airflow] Migrate mediawiki geoeditors druid loading job
Resolved	Snwachukwu	T334104 [Airflow] Migrate pageview-related Druid loading Oozie jobs
Resolved	Milimetric	T334105 [Airflow] Migrate virtualpageviews Druid loading Oozie jobs
Resolved	mforns	T334106 [Airflow] Migrate webrequest sampled 128 druid loading jobs
Resolved	mforns	T336184 Migrate banner-activity jobs from Oozie to Airflow

Event Timeline

JArguello-WMF created this task.Dec 5 2022, 4:54 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 5 2022, 4:54 PM

JArguello-WMF moved this task from To be discussed /To be estimated to To be prioritised on the Data Pipelines board.Dec 5 2022, 4:57 PM

@mforns / @JAllemandou can you please fill in the details of this one to make it easier for folks to size? Thank you!

• EChetty moved this task from To be prioritised to To be discussed /To be estimated on the Data Pipelines board.Jan 9 2023, 3:33 PM

mforns updated the task description. (Show Details)Jan 9 2023, 7:02 PM

• EChetty assigned this task to mforns.Jan 16 2023, 2:42 PM

• EChetty moved this task from To be discussed /To be estimated to Sprint 07 on the Data Pipelines board.

• EChetty edited projects, added Data Pipelines (Sprint 07); removed Data Pipelines.

• EChetty moved this task from Ready to Next Up on the Data Pipelines (Sprint 07) board.Jan 16 2023, 4:13 PM

• EChetty set the point value for this task to 9.Jan 16 2023, 4:18 PM

• EChetty moved this task from Next Up to In Progress on the Data Pipelines (Sprint 07) board.Jan 17 2023, 5:10 PM

Waiting for the Airflow upgrade.

Change 886114 had a related patch set uploaded (by Mforns; author: Mforns):

[analytics/refinery/source@master] Support snapshot partitioning in HiveToDruid and DataFrameToDruid

https://gerrit.wikimedia.org/r/886114

gerritbot added a project: Patch-For-Review.Feb 2 2023, 5:16 PM

• EChetty moved this task from Sprint 07 to Sprint 08 on the Data Pipelines board.Feb 7 2023, 12:26 PM

• EChetty edited projects, added Data Pipelines (Sprint 08); removed Data Pipelines (Sprint 07).

• EChetty moved this task from Ready to In Progress on the Data Pipelines (Sprint 08) board.

Change 886114 merged by jenkins-bot:

[analytics/refinery/source@master] Support snapshot partitioning in HiveToDruid and DataFrameToDruid

https://gerrit.wikimedia.org/r/886114

Maintenance_bot removed a project: Patch-For-Review.Feb 8 2023, 6:31 PM

JArguello-WMF edited projects, added Data Pipelines (Sprint 11); removed Data Pipelines (Sprint 08).Feb 20 2023, 5:22 PM

JArguello-WMF moved this task from Next Up to In Progress on the Data Pipelines (Sprint 11) board.Feb 20 2023, 5:22 PM

JArguello-WMF moved this task from In Progress to Blocked/Paused on the Data Pipelines (Sprint 11) board.Feb 23 2023, 5:07 PM

JArguello-WMF moved this task from Blocked/Paused to In Progress on the Data Pipelines (Sprint 11) board.Mar 2 2023, 5:49 PM

JArguello-WMF edited projects, added Data Pipelines (sprint 10); removed Data Pipelines (Sprint 11).Mar 13 2023, 1:57 PM

JArguello-WMF moved this task from Ready to In Progress on the Data Pipelines (sprint 10) board.

JArguello-WMF reassigned this task from mforns to Milimetric.Mar 13 2023, 4:47 PM

note to self, look at T331892: Move eventlogging_to_druid_ jobs to airflow in order to rely on cluster spark mechanism instead of client

This is temporarily in review to get opinions on the way I handled the delayed daily timetable interacting with our datasets idea. It's kind of hard coded but I think simpler than a more flexible approach. Let me know what you think. (assigned to Sandra but anyone is welcome to comment)

https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/217

lbowmaker mentioned this in T331892: Move eventlogging_to_druid_ jobs to airflow in order to rely on cluster spark mechanism instead of client.Mar 22 2023, 5:41 PM

JArguello-WMF reassigned this task from Milimetric to mforns.Mar 27 2023, 4:06 PM

JArguello-WMF added a subscriber: Milimetric.

BTullis mentioned this in T331265: Investigate DB connection issues faced from airflow on an-launcher1002.Mar 30 2023, 1:05 PM

BTullis subscribed.

Change 904609 had a related patch set uploaded (by Mforns; author: Mforns):

[operations/puppet@production] modules::profile::manifests::airflow.pp: add plugins_folder path

https://gerrit.wikimedia.org/r/904609

gerritbot added a project: Patch-For-Review.Mar 30 2023, 6:24 PM

JArguello-WMF edited projects, added Data Pipelines (Sprint 11); removed Data Pipelines (sprint 10).Mar 31 2023, 1:56 PM

JArguello-WMF moved this task from In Progress to In Review on the Data Pipelines (Sprint 11) board.

This task was underestimated, and we should split it up in subtasks, one per Druid job.
Will do so and set all of those tasks as children of this one.

mforns moved this task from In Review to Next Up on the Data Pipelines (Sprint 11) board.Apr 5 2023, 2:30 PM

mforns moved this task from Next Up to In Progress on the Data Pipelines (Sprint 11) board.Apr 5 2023, 2:52 PM

mforns moved this task from In Progress to Next Up on the Data Pipelines (Sprint 11) board.Apr 5 2023, 2:56 PM

mforns removed mforns as the assignee of this task.Apr 5 2023, 3:07 PM

mforns changed the point value for this task from 9 to 0.

mforns renamed this task from [Migration] Oozie jobs for Druid to [Airflow] Migrate Oozie jobs for Druid - Parent task.Apr 5 2023, 3:18 PM

mforns renamed this task from [Airflow] Migrate Oozie jobs for Druid - Parent task to [Airflow] Migrate Druid loading Oozie jobs - Parent task.

Change 904609 merged by Stevemunene:

[operations/puppet@production] modules::profile::manifests::airflow.pp: add plugins_folder path

https://gerrit.wikimedia.org/r/904609

Maintenance_bot removed a project: Patch-For-Review.Apr 6 2023, 7:10 PM

JArguello-WMF closed subtask T334095: [Airflow] Migrate Netflow Druid loading jobs as Resolved.Apr 13 2023, 1:23 PM