Page MenuHomePhabricator
Create Task
Maniphest T282035

Catalog, Categorize, and Templetize existing scheduled workflows
Closed, ResolvedPublic
Actions

Description

The output of this task should be a catalog of all existing scheduled jobs that Analytics is responsible for, along with other jobs whose owners are interested in migrating to AirFlow. A first-pass at such a catalog is in this document, and the final output doesn't have to be much more precise, we just need a rough idea of what kinds of jobs we have so we can make good templates and implement them cleanly in the new system:

https://docs.google.com/spreadsheets/d/1lfK5Idteh6zPSlCWyH34FJCl_Lcm8401Wm59Jgk-7wM/edit#gid=0

  • Oozie
  • Reportupdater
  • timers/cron
  • refine

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT282033 Airflow collaborations
 ResolvedNoneT282035 Catalog, Categorize, and Templetize existing scheduled workflows

Event Timeline

Milimetric created this task.May 5 2021, 6:15 PM
Ottomata moved this task from Incoming to Smart Tools for Better Data on the Analytics board.May 6 2021, 5:07 PM
Milimetric triaged this task as High priority.May 10 2021, 4:16 PM
• Pchelolo added a subscriber: • Pchelolo.May 10 2021, 4:19 PM

I can't edit the docs, but please don't forget about how we are using job queue as a scheduler for newcomerTaskCacheRefreshJob and fetchGoogleCloudAnnotationsJob

• AMooney moved this task from Inbox to Tracking/Watching on the Platform Engineering board.May 11 2021, 9:16 PM
mforns moved this task from Smart Tools for Better Data to Airflow on the Analytics board.Jun 25 2021, 4:39 PM
mforns edited projects, added Data-Engineering, Data Pipelines; removed Analytics.Nov 5 2021, 5:33 PM
odimitrijevic lowered the priority of this task from High to Medium.Nov 8 2021, 4:56 PM
odimitrijevic moved this task from Incoming to Transform on the Data-Engineering board.
JAllemandou updated the task description. (Show Details)Feb 1 2022, 8:54 AM
EChetty moved this task from Backlog to To be discussed on the Data Pipelines board.Oct 14 2022, 8:51 AM
EChetty moved this task from To be discussed to To be prioritised on the Data Pipelines board.Nov 20 2022, 5:49 PM
mforns updated the task description. (Show Details)Nov 21 2022, 3:23 PM
EChetty closed this task as Resolved.Nov 28 2022, 8:20 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL