Present "Notebooks in Airflow" solution to PA and discuss ownership of different steps
Open, Needs TriagePublic
Actions

Assigned To

Authored By

	mforns
	Dec 14 2022, 3:36 PM

Description

Ownership Map -> What we will own vs What others will own
- Show idea (NotebookOperator in Airflow DAG, using single conda env automatically packaged by CI) to Product Analytics and ask if they would like that.
- Discuss who would take care of writing DAGs, testing DAGs, reviewing DAG code, merging, deploying Airflow, receiving alerts, troubleshooting failed DAGs, updating the notebooks conda env with new libraries, etc.

Details

Other Assignee: mforns

Related Objects
Search...

Status	Assigned	Task
Open	mpopov	T316049 Unify all Product Analytics ETL jobs
Open	None	T322532 Notebook Scheduler for Product Analytics
Resolved	mforns	T322534 Spike: Product Analytics ETL options - Timebox 1 Sprint.
Open	lbowmaker	T325181 Present "Notebooks in Airflow" solution to PA and discuss ownership of different steps

Event Timeline

mforns created this task.Dec 14 2022, 3:36 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 14 2022, 3:36 PM

mforns added a parent task: T322534: Spike: Product Analytics ETL options - Timebox 1 Sprint..Dec 14 2022, 3:36 PM

mforns mentioned this in T322534: Spike: Product Analytics ETL options - Timebox 1 Sprint..Dec 14 2022, 4:05 PM

• EChetty moved this task from Backlog to To be discussed /To be estimated on the Data Pipelines board.Jan 6 2023, 12:57 PM

• EChetty moved this task from To be discussed /To be estimated to To be prioritised on the Data Pipelines board.Jan 17 2023, 11:45 AM

• EChetty claimed this task.Jan 17 2023, 11:48 AM

• EChetty triaged this task as High priority.

• EChetty updated Other Assignee, added: mforns.

• EChetty moved this task from To be prioritised to Discussed (Radar) on the Data Pipelines board.

Removing inactive assignee (please do so as part of team offboarding!).

mpopov added a project: Product-Analytics.Mar 27 2023, 5:23 PM

mpopov moved this task from Triage to Tracking on the Product-Analytics board.

The description probably needs updated since the proposal we're going with covers a lot of ownership questions, but there are still open ones around what notebook DAGs look like, where artifacts are stored, how notebooks are versioned, private repos, etc.

JArguello-WMF edited projects, added Data Engineering and Event Platform Team; removed Data Pipelines.Jun 30 2023, 5:43 PM

JArguello-WMF moved this task from Data Eng Backlog to Radar (External Teams) on the Data Engineering and Event Platform Team board.

lbowmaker edited projects, added Data-Engineering; removed Data Engineering and Event Platform Team.Nov 10 2023, 2:22 PM

lbowmaker moved this task from Incoming (new tickets) to Radar (External Teams) on the Data-Engineering board.

nshahquinn-wmf subscribed.Feb 26 2024, 8:47 PM