Migrate the movement_metrics ETL jobs to Airflow
Open, HighPublic
Actions

Assigned To

Authored By

	mpopov
	Mar 27 2023, 8:12 PM

Description

Migrate the current ETLs in analytics/wmf-product/jobs/movement_metrics (scheduled via systemd timer in puppet) to Product Analytics Airflow.

As part of this, we should:

rebuild each of the tables from scratch. In most cases, it shouldn't make much difference, but it will address T358684 and possibly some other small deviations. (This does not mean we will recalculate all the metric values that have been derived from these tables over the years.)
do the rebuilding in Iceberg format
move each table to the appropriate one of the new "wmf_*" databases.

Related Objects
Search...

Status	Assigned	Task
Open	nshahquinn-wmf	T359207 Improve the delivery of the movement movements (SDS 2.6.2)
Invalid	nshahquinn-wmf	T359697 Document tables managed by the movement_metrics ETL job
Open	None	T359700 Ensure that movement metrics Superset dashboards are functional and performant
Open	nshahquinn-wmf	T333225 Migrate the movement_metrics ETL jobs to Airflow
Declined	None	T308077 Add partitioning to editors tables
Open	nshahquinn-wmf	T314541 Update the active_editors table with an Airflow job
Resolved	nshahquinn-wmf	T359691 Remove any unnecessary intermediate tables managed by the movement_metrics ETL job
Resolved	nshahquinn-wmf	T359690 Determine if any intermediate tables managed by the movement_metrics ETL job can be removed
Resolved	nshahquinn-wmf	T359687 Determine which metrics and charts can be eliminated from the movement-metrics repository
Resolved	nshahquinn-wmf	T359696 Study Airflow and do any necessary set-up
Resolved	brennen	T360956 Create new GitLab project group: movement-insights
Resolved	nshahquinn-wmf	T360491 Decide how to handle metric corrections
Open	nshahquinn-wmf	T362593 Update the content_interactions table with an Airflow job
Open	nshahquinn-wmf	T362594 Update the editor_month table with an Airflow job
Open	nshahquinn-wmf	T362595 Update the new_editors table with an Airflow job

Event Timeline

mpopov triaged this task as Medium priority.Mar 27 2023, 8:12 PM

mpopov created this task.

mpopov added subtasks: T308077: Add partitioning to editors tables, T314541: Update the active_editors table with an Airflow job.

mpopov moved this task from Triage to Blocked on the Product-Analytics board.

mpopov added a project: Movement-Insights.Nov 8 2023, 12:56 PM

OSefu-WMF moved this task from Incoming to Watching on the Movement-Insights board.Nov 8 2023, 9:55 PM

Now owned by Movement Insights

Oops!

nshahquinn-wmf closed subtask T308077: Add partitioning to editors tables as Declined.Mar 5 2024, 6:36 PM

nshahquinn-wmf renamed this task from Product Analytics ETL Migration: Movement metrics to Migrate the movement_metrics ETL jobs to Airflow.Mar 5 2024, 7:40 PM

nshahquinn-wmf claimed this task.

nshahquinn-wmf added a parent task: T359207: Improve the delivery of the movement movements (SDS 2.6.2).

nshahquinn-wmf merged a task: T307598: Modify ETL Update queries to INSERT OVERWRITE.Mar 5 2024, 7:52 PM

nshahquinn-wmf added a subscriber: Aklapper.

nshahquinn-wmf removed a subtask: T359690: Determine if any intermediate tables managed by the movement_metrics ETL job can be removed.Mar 9 2024, 1:01 AM

nshahquinn-wmf removed subscribers: Iflorez, mpopov.Mar 9 2024, 1:44 AM

nshahquinn-wmf mentioned this in T359700: Ensure that movement metrics Superset dashboards are functional and performant.Mar 9 2024, 2:26 AM

nshahquinn-wmf added a parent task: T359700: Ensure that movement metrics Superset dashboards are functional and performant.Mar 9 2024, 2:28 AM

nshahquinn-wmf mentioned this in T358684: Active editor queries incorrectly calculate global registration date.Mar 19 2024, 9:53 PM

nshahquinn-wmf updated the task description. (Show Details)Mar 19 2024, 9:56 PM

I've created https://gitlab.wikimedia.org/repos/movement-insights/sql to hold the SQL for these jobs.

nshahquinn-wmf updated the task description. (Show Details)Apr 6 2024, 3:31 AM