Page MenuHomePhabricator

Spike: POC of refine with airflow
Closed, ResolvedPublic

Description

We want to try Airflow as an easier scheduling alternative, let's do a poc for the refine workflow

Event Timeline

fdans moved this task from Incoming to Smart Tools for Better Data on the Analytics board.

Change 582114 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] [WIP] Airflow aqs-data-extraction example

https://gerrit.wikimedia.org/r/582114

mforns added a project: Analytics-Kanban.
mforns moved this task from Next Up to In Progress on the Analytics-Kanban board.

Change 597623 had a related patch set uploaded (by Mforns; owner: Mforns):
[analytics/refinery@master] POC Airflow Refine

https://gerrit.wikimedia.org/r/597623

I tested the code above in an-launcher1001 with the analytics user successfully!
Note I used Airflow's sequentialExecutor, because I was using an Airflow sqlite install.
I will try to install MySQL in that machine with the help of an ops :],
to be able to test Airflow Refine in parallel.
But looks good so far!

Change 597623 abandoned by Mforns:
[analytics/refinery@master] POC Airflow Refine

Reason:
Archiving. Was never intended for merging.

https://gerrit.wikimedia.org/r/597623