Investigate and PoC scripts orchestration with airflow
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Add an Airflow DAG for model training and etl. | mediawiki/services/similar-users | main | +379 -3 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T259471 Sockpuppet detection API [low effort] | |||
| Resolved | None | T265722 New Service Request: Sockpuppet Detection | |||
| Resolved | odimitrijevic | T282033 Airflow collaborations | |||
| Resolved | gmodena | T270613 [SPIKE] can we orchestrate the Similarusers pipeline with airflow? |
Event Timeline
Comment Actions
We have an example of basic airflow dag at https://gerrit.wikimedia.org/r/c/mediawiki/services/similar-users/+/654485. This is by all means nothing production ready, but rather a PoC to help with reasoning about data movements.
Comment Actions
Change 654485 had a related patch set uploaded (by Gmodena; owner: Gmodena):
[mediawiki/services/similar-users@main] Add an Airflow DAG for etl.