Maniphest T300029

Low Risk Oozie Migration: session length
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• EChetty
	Jan 25 2022, 11:55 AM

Tags

Referenced Files

None

Subscribers

Description

Goal:
Migrate the session length jobs to Airflow:

Job Details:

Input	Processing	Output
Hive Table	Hive	Hive Table

Success Criteria:

Have the 1 daily Jobs Migrated (SLA 30 Hours)

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Make caching mechanisms thread ready	analytics/refinery/source	master	+300 -289
	Fix Airflow job session length daily	analytics/refinery	master	+9 -4

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		• odimitrijevic	T282033 Airflow collaborations
		Resolved		• odimitrijevic	T271429 Replace Oozie with better workflow scheduler
		Resolved		• odimitrijevic	T299074 Migrate Oozie jobs to Airflow
		Resolved		Antoine_Quhen	T300029 Low Risk Oozie Migration: session length

Event Timeline

• EChetty created this task.Jan 25 2022, 11:55 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 25 2022, 11:55 AM

• EChetty added a parent task: T299074: Migrate Oozie jobs to Airflow.Jan 25 2022, 12:00 PM

JAllemandou updated the task description. (Show Details)Jan 25 2022, 1:07 PM

• odimitrijevic moved this task from Incoming (new tickets) to Transform on the Data-Engineering board.Feb 1 2022, 11:10 PM

• EChetty moved this task from Backlog to Discussed (Radar) on the Data Pipelines board.Feb 8 2022, 11:42 AM

Antoine_Quhen claimed this task.Mar 8 2022, 5:04 PM

Antoine_Quhen added a project: Data-Engineering-Kanban.Mar 9 2022, 6:34 PM

Antoine_Quhen moved this task from Next Up to In Progress on the Data-Engineering-Kanban board.Mar 9 2022, 8:11 PM

Antoine_Quhen moved this task from In Progress to In Code Review on the Data-Engineering-Kanban board.

Change 769512 had a related patch set uploaded (by Aqu; author: Aqu):

[analytics/refinery@master] Migrate session_length/daily from Oozie to Airflow

https://gerrit.wikimedia.org/r/769512

gerritbot added a project: Patch-For-Review.Mar 9 2022, 8:19 PM

Antoine_Quhen moved this task from Discussed (Radar) to In Review on the Data Pipelines board.Mar 9 2022, 8:40 PM

Antoine_Quhen moved this task from In Code Review to Ready to Deploy on the Data-Engineering-Kanban board.Mar 16 2022, 4:05 PM

https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/33

Antoine_Quhen moved this task from In Review to Done on the Data Pipelines board.Mar 16 2022, 6:16 PM

Antoine_Quhen moved this task from Ready to Deploy to Done on the Data-Engineering-Kanban board.

Antoine_Quhen moved this task from Done to In Review on the Data Pipelines board.Mar 17 2022, 9:55 PM

Antoine_Quhen moved this task from Done to In Progress on the Data-Engineering-Kanban board.

Change 771730 had a related patch set uploaded (by Aqu; author: Aqu):

[analytics/refinery@master] Fix Airflow job session length daily

https://gerrit.wikimedia.org/r/771730

• EChetty moved this task from In Review to Done on the Data Pipelines board.Mar 21 2022, 3:55 PM

Change 771730 abandoned by Aqu:

[analytics/refinery@master] Fix Airflow job session length daily

Reason:

Dropped in profit of https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/772027

https://gerrit.wikimedia.org/r/771730

Antoine_Quhen moved this task from In Progress to In Code Review on the Data-Engineering-Kanban board.Mar 28 2022, 2:11 PM

Antoine_Quhen moved this task from In Code Review to Ready to Deploy on the Data-Engineering-Kanban board.

Change 772027 had a related patch set uploaded (by Aqu; author: Aqu):

[analytics/refinery/source@master] Fix: Prevent empty normalized host

https://gerrit.wikimedia.org/r/772027

Change 772027 merged by jenkins-bot:

[analytics/refinery/source@master] Make caching mechanisms thread ready

https://gerrit.wikimedia.org/r/772027

Antoine_Quhen moved this task from Ready to Deploy to Done on the Data-Engineering-Kanban board.Apr 28 2022, 3:44 PM

JArguello-WMF closed this task as Resolved.May 31 2022, 3:29 PM