Page MenuHomePhabricator

[SPIKE] can we orchestrate the Similarusers pipeline with airflow?
Closed, ResolvedPublic

Description

Investigate and PoC scripts orchestration with airflow

Event Timeline

We have an example of basic airflow dag at https://gerrit.wikimedia.org/r/c/mediawiki/services/similar-users/+/654485. This is by all means nothing production ready, but rather a PoC to help with reasoning about data movements.

Change 654485 had a related patch set uploaded (by Gmodena; owner: Gmodena):
[mediawiki/services/similar-users@main] Add an Airflow DAG for etl.

https://gerrit.wikimedia.org/r/654485

gmodena renamed this task from [SPIKE] can we orchestrate (parts of) ETL with airflow? to [SPIKE] can we orchestrate the Similarusers pipeline with airflow?.May 6 2021, 5:57 AM