Maniphest T336184

Migrate banner-activity jobs from Oozie to Airflow
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	lbowmaker
	May 8 2023, 4:37 PM

Tags

Referenced Files

None

Subscribers

Description

Migrate banner-activity (daily and monthly) jobs from Oozie to Airflow.

Done is:

Output from Oozie and Airflow jobs is the same
Airflow jobs runs successfully
Oozie jobs are turned off

https://github.com/wikimedia/analytics-refinery/tree/master/oozie/banner_activity/druid

Details

	Subject	Repo	Branch	Lines +/-
	Migrate banner activity druid loading to Spark3 and Airflow	analytics/refinery	master	+190 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		JArguello-WMF	T324485 [Airflow] Migrate Druid loading Oozie jobs - Parent task
		Resolved		mforns	T336184 Migrate banner-activity jobs from Oozie to Airflow

Event Timeline

lbowmaker created this task.May 8 2023, 4:37 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 8 2023, 4:37 PM

lbowmaker added a subscriber: AndyRussG.May 8 2023, 7:06 PM

lbowmaker moved this task from To be discussed /To be estimated to Sprint 14 on the Data Pipelines board.May 15 2023, 4:03 PM

lbowmaker edited projects, added Data Pipelines (Sprint 14); removed Data Pipelines.

mforns claimed this task.May 15 2023, 4:03 PM

mforns moved this task from Next Up to In Progress on the Data Pipelines (Sprint 14) board.

JArguello-WMF set the point value for this task to 3.May 15 2023, 4:09 PM

mforns added a parent task: T324485: [Airflow] Migrate Druid loading Oozie jobs - Parent task.May 15 2023, 4:47 PM

Change 919883 had a related patch set uploaded (by Mforns; author: Mforns):

[analytics/refinery@master] Migrate banner activity druid loading to Spark3 and Airflow

https://gerrit.wikimedia.org/r/919883

gerritbot added a project: Patch-For-Review.May 15 2023, 7:46 PM

Here's the MR for the new Airflow DAG: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/387

mforns moved this task from In Progress to In Review on the Data Pipelines (Sprint 14) board.May 16 2023, 8:08 AM

Change 919883 merged by Joal:

[analytics/refinery@master] Migrate banner activity druid loading to Spark3 and Airflow

https://gerrit.wikimedia.org/r/919883

Maintenance_bot removed a project: Patch-For-Review.May 16 2023, 12:10 PM

JAllemandou moved this task from In Review to Done on the Data Pipelines (Sprint 14) board.May 16 2023, 3:58 PM

JArguello-WMF closed this task as Resolved.May 19 2023, 4:02 PM