Maniphest T300026

Low Risk Oozie Migration: wikidata_json_entity
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• EChetty
	Jan 25 2022, 11:54 AM

Tags

Referenced Files

None

Subscribers

Description

Goal:
Migrate the wikidata_json_entity jobs to Airflow:

Job Details:

Input	Processing	Output
Dumps	Spark	Hive Table

Success Criteria:

Have the 1 Weekly Job Migrated (14 Days)

NOTE: this job is extremely similar to the new job @Snwachukwu is writing, and also to mediawiki/wikitext 2 jobs - We should take advantage of that and build a DAG factory for dumps processing.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		odimitrijevic	T282033 Airflow collaborations
		Resolved		odimitrijevic	T271429 Replace Oozie with better workflow scheduler
		Resolved		odimitrijevic	T299074 Migrate Oozie jobs to Airflow
		Resolved		Snwachukwu	T300026 Low Risk Oozie Migration: wikidata_json_entity

Event Timeline

• EChetty created this task.Jan 25 2022, 11:54 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 25 2022, 11:54 AM

• EChetty added a parent task: T299074: Migrate Oozie jobs to Airflow.Jan 25 2022, 12:01 PM

JAllemandou updated the task description. (Show Details)Jan 25 2022, 1:18 PM

JAllemandou added a subscriber: Snwachukwu.

odimitrijevic moved this task from Incoming (new tickets) to Transform on the Data-Engineering board.Feb 1 2022, 11:11 PM

• EChetty moved this task from Backlog to Estimated on the Data Pipelines board.Feb 7 2022, 4:45 PM

Snwachukwu claimed this task.Feb 7 2022, 4:45 PM

Snwachukwu added a project: Data-Engineering-Kanban.Feb 8 2022, 5:05 PM

Snwachukwu moved this task from Next Up to In Progress on the Data-Engineering-Kanban board.

Snwachukwu moved this task from In Progress to In Code Review on the Data-Engineering-Kanban board.Feb 23 2022, 5:05 PM

Snwachukwu moved this task from Estimated to In Review on the Data Pipelines board.Feb 23 2022, 5:08 PM

Snwachukwu moved this task from In Code Review to Ready to Deploy on the Data-Engineering-Kanban board.Feb 28 2022, 4:08 PM

mforns moved this task from In Review to Done on the Data Pipelines board.Feb 28 2022, 4:39 PM

Ottomata moved this task from Ready to Deploy to Done on the Data-Engineering-Kanban board.Mar 8 2022, 5:05 PM

JArguello-WMF closed this task as Resolved.May 31 2022, 3:37 PM