Maniphest T300344

Low Risk Oozie Migration: Mediawiki History Dumps
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	JAllemandou
	Jan 28 2022, 9:45 AM

Tags

Referenced Files

None

Subscribers

• odimitrijevic

Description

Goal:
Migrate the mediawiki-history-dumps jobs to Airflow (Spark job):

Job Details:

Input	Processing	Output
Hive Table	Spark	Archive

Success Criteria:

Have the 1 monthly Job Migrated (SLA 35 days)

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		• odimitrijevic	T282033 Airflow collaborations
		Resolved		• odimitrijevic	T271429 Replace Oozie with better workflow scheduler
		Resolved		• odimitrijevic	T299074 Migrate Oozie jobs to Airflow
		Resolved		xcollazo	T300344 Low Risk Oozie Migration: Mediawiki History Dumps

Event Timeline

JAllemandou created this task.Jan 28 2022, 9:45 AM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptJan 28 2022, 9:45 AM

• odimitrijevic moved this task from Incoming (new tickets) to Transform on the Data-Engineering board.Feb 1 2022, 11:10 PM

• EChetty moved this task from Backlog to Epics on the Data Pipelines board.Feb 8 2022, 11:42 AM

JAllemandou moved this task from Epics to Discussed (Radar) on the Data Pipelines board.Apr 25 2022, 3:23 PM

JArguello-WMF removed a project: Epic.Apr 25 2022, 3:23 PM

JArguello-WMF removed a project: Data-Engineering-Kanban.Apr 27 2022, 1:21 PM

• EChetty edited projects, added Data-Engineering-Planning; removed Data-Engineering.Jun 30 2022, 3:05 PM

• EChetty set the point value for this task to 3.Aug 16 2022, 3:15 PM

• EChetty moved this task from Discussed (Radar) to Sprint 00 on the Data Pipelines board.Aug 16 2022, 3:17 PM

• EChetty edited projects, added Data Pipelines (Sprint 00); removed Data Pipelines.

• EChetty moved this task from Ready to Next Up on the Data Pipelines (Sprint 00) board.Aug 19 2022, 11:20 AM

xcollazo claimed this task.Aug 23 2022, 2:15 PM

• EChetty moved this task from Next Up to In Progress on the Data Pipelines (Sprint 00) board.Aug 23 2022, 4:11 PM

xcollazo mentioned this in T316371: Migrate Mediawiki History Dumps to Spark3.Aug 26 2022, 4:21 PM

xcollazo moved this task from In Progress to In Review on the Data Pipelines (Sprint 00) board.

xcollazo moved this task from In Review to Ready to Deploy on the Data Pipelines (Sprint 00) board.Aug 26 2022, 6:44 PM

Delivered via:
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/127
and
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/128

Opened follow up task T316371 to migrate to Spark3.

Verified working on prod. Closing.

• EChetty moved this task from Ready to Deploy to Done on the Data Pipelines (Sprint 00) board.Sep 5 2022, 3:47 PM