⚓ T241246 Spike: POC of refine with airflow

	Subject	Repo	Branch	Lines +/-
	POC Airflow Refine	analytics/refinery	master	+554 -0
	[WIP] Airflow aqs-data-extraction example	analytics/refinery	master	+136 -0

Status	Assigned	Task
Resolved	• odimitrijevic	T282033 Airflow collaborations
Resolved	• odimitrijevic	T271429 Replace Oozie with better workflow scheduler
Resolved	mforns	T241246 Spike: POC of refine with airflow
Resolved	mforns	T248328 Test aqs_hourly job from Airflow testing instance

• Nuria created this task.Dec 20 2019, 3:44 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 20 2019, 3:44 PM

• fdans triaged this task as High priority.Dec 23 2019, 4:25 PM

• fdans moved this task from Incoming to Smart Tools for Better Data on the Analytics board.

kzimmerman subscribed.Dec 23 2019, 8:44 PM

SNowick_WMF subscribed.Jan 7 2020, 6:34 PM

mpopov subscribed.Jan 7 2020, 7:01 PM

Ottomata subscribed.Jan 21 2020, 5:44 PM

elukey subscribed.Feb 4 2020, 1:07 PM

mforns subscribed.Mar 5 2020, 6:00 PM

Change 582114 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] [WIP] Airflow aqs-data-extraction example

https://gerrit.wikimedia.org/r/582114

gerritbot added a project: Patch-For-Review.Mar 20 2020, 5:05 PM

mforns mentioned this in T248328: Test aqs_hourly job from Airflow testing instance.Mar 23 2020, 5:06 PM

mforns added a subtask: T248328: Test aqs_hourly job from Airflow testing instance.

• Nuria closed subtask T248328: Test aqs_hourly job from Airflow testing instance as Resolved.Apr 27 2020, 8:38 PM

mforns claimed this task.Apr 30 2020, 7:23 PM

mforns added a project: Analytics-Kanban.

mforns moved this task from Next Up to In Progress on the Analytics-Kanban board.

Change 597623 had a related patch set uploaded (by Mforns; owner: Mforns):
[analytics/refinery@master] POC Airflow Refine

https://gerrit.wikimedia.org/r/597623

I tested the code above in an-launcher1001 with the analytics user successfully!
Note I used Airflow's sequentialExecutor, because I was using an Airflow sqlite install.
I will try to install MySQL in that machine with the help of an ops :],
to be able to test Airflow Refine in parallel.
But looks good so far!