Page MenuHomePhabricator
Create Task
Maniphest T271429

Replace Oozie with better workflow scheduler
Closed, ResolvedPublic
Actions

Description

Oozie is hard to learn and a poor fit for many of our use cases. As a result, we use it alongside a variety of other workflow schedulers. We should replace it with a better tool that can manage all of our workflows.

Apache Airflow is the leading candidate. More details at wikitech:Analytics/Systems/Cluster/Workflow management tools study

Tentatively planned for Apr-Jun 2021.

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedodimitrijevicT282033 Airflow collaborations
 ResolvedodimitrijevicT271429 Replace Oozie with better workflow scheduler
 ResolvedmfornsT241246 Spike: POC of refine with airflow
 ResolvedmfornsT248328 Test aqs_hourly job from Airflow testing instance
 ResolvedNoneT217059 Spike [2019-2020 work] Oozie Replacement. Airflow Study / Argo Study
 ResolvedOttomataT272973 Generalize the current Airflow puppet/scap code to deploy a dedicated Analytics instance
 ResolvedOttomataT277012 Create a debian package for Apache Airflow
 DuplicateNoneT284172 [SPIKE] analytics-airflow jobs development
 ResolvedmfornsT285692 Write a job entirely in Airflow with spark and/or sparkSQL
 ResolvedmfornsT285695 Finish document for the Airflow POC
 ResolvedodimitrijevicT299074 Migrate Oozie jobs to Airflow
 ResolvedAntoine_QuhenT299398 Low Risk Oozie Migration: aqs
 ResolvedAntoine_QuhenT300029 Low Risk Oozie Migration: session length
 ResolvedSnwachukwuT300028 Low Risk Oozie Migration: APIs
 DuplicateNoneT300027 Low Risk Oozie Migration: aqs
 ResolvedSnwachukwuT300026 Low Risk Oozie Migration: wikidata_json_entity
 Resolved• NOkafor-WMFT300025 Low Risk Oozie Migration: interlanguage
 ResolvedAntoine_QuhenT300023 Low Risk Oozie Migration: wikidata_item_page_link
 ResolvedSnwachukwuT300021 Low Risk Oozie Migration: 4 wikidata metrics jobs
 ResolvedSnwachukwuT306177 Low RIsk Ozzie Migration: Wikidata CoEditor metric job
 ResolvedAntoine_QuhenT300039 Define and implement archiving for Airflow
 Resolved• ntsakoT300282 Low Risk Oozie Migration: Mediawiki Geoeditors Monthly
 ResolvedxcollazoT300344 Low Risk Oozie Migration: Mediawiki History Dumps

Event Timeline

nshahquinn-wmf created this task.Jan 7 2021, 3:42 PM
nshahquinn-wmf moved this task from Triage to Tracking on the Product-Analytics board.
nshahquinn-wmf added subtasks: T241246: Spike: POC of refine with airflow, T217059: Spike [2019-2020 work] Oozie Replacement. Airflow Study / Argo Study.Jan 7 2021, 3:47 PM
nshahquinn-wmf added a project: Epic.
JAllemandou subscribed.Jan 7 2021, 4:48 PM
• fdans triaged this task as High priority.Jan 11 2021, 5:04 PM
• fdans moved this task from Incoming to Smart Tools for Better Data on the Analytics board.
Ottomata added subscribers: mforns, • razzi, Milimetric.May 19 2021, 8:15 PM
Ottomata changed the status of subtask T217059: Spike [2019-2020 work] Oozie Replacement. Airflow Study / Argo Study from Duplicate to Resolved.May 19 2021, 8:18 PM
Ottomata added a subtask: T272973: Generalize the current Airflow puppet/scap code to deploy a dedicated Analytics instance.
Ottomata added a parent task: T282033: Airflow collaborations.Jun 3 2021, 5:27 PM
mforns moved this task from Smart Tools for Better Data to Airflow on the Analytics board.Jun 25 2021, 4:35 PM
nshahquinn-wmf mentioned this in T287306: Create an ETL job to aggregate Content Translation metrics for dashboarding.Jul 23 2021, 9:43 PM
Ottomata closed subtask T272973: Generalize the current Airflow puppet/scap code to deploy a dedicated Analytics instance as Resolved.Jul 27 2021, 7:20 PM
mforns edited projects, added Data-Engineering, Data Pipelines; removed Analytics.Nov 5 2021, 5:01 PM
mforns moved this task from Backlog to Epics on the Data Pipelines board.Nov 5 2021, 9:10 PM
mforns moved this task from Incoming (new tickets) to Transform on the Data-Engineering board.Nov 8 2021, 4:57 PM
odimitrijevic added a subtask: T299074: Migrate Oozie jobs to Airflow.Jan 17 2022, 4:51 PM
JArguello-WMF moved this task from Transform to Event Platform Backlog on the Data-Engineering board.Jun 29 2023, 11:21 PM
JArguello-WMF added a project: Data Engineering and Event Platform Team.Jun 30 2023, 5:31 PM
JArguello-WMF moved this task from Data Eng Backlog to Parent Tasks/Epics on the Data Engineering and Event Platform Team board.Jun 30 2023, 5:38 PM
odimitrijevic closed this task as Resolved.Jul 6 2023, 1:21 PM
odimitrijevic claimed this task.
odimitrijevic closed subtask T299074: Migrate Oozie jobs to Airflow as Resolved.
odimitrijevic subscribed.

So gratifying to be able to be closing this task!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL