Deprecate old mobile datasets
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	JAllemandou
	Feb 9 2023, 5:27 PM

Description

We plan on deprecating 4 tables:

mobile_apps_uniques_daily not updated since 2021-02-18
mobile_apps_uniques_monthly still active (on 2023-05-01)
mobile_apps_session_metrics not updated since 2018-03-20
mobile_apps_session_metrics_by_os still active (on 2023-05-07)

Our plan is to:

Pause the computation jobs (there's only 2 active, one in Oozie, one in Airflow)
Create MR to remove the Airflow DAG https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/379
Create a patch to remove the refinery-source Spark code https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/919357
Drop the hive table but keep the data (we can recreate the tables easily)
Wait for 30 days, and in case nobody has asked about the data...
Remove the code from refine and airflow-dags
Delete the old data altogether

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Remove queries for deprecated mobile_apps jobs	analytics/refinery	master	+0 -117
	Remove deprecated code for AppSessionMetrics	analytics/refinery/source	master	+0 -521

Customize query in gerrit

Related Objects

Mentioned In: T329978: Delete empty tables unique_devices_*_wide_*
T327458: Document Traffic Datasets in Datahub
Mentioned Here: T329978: Delete empty tables unique_devices_*_wide_*

Event Timeline

JAllemandou created this task.Feb 9 2023, 5:27 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 9 2023, 5:27 PM

JAllemandou updated the task description. (Show Details)Feb 9 2023, 5:28 PM

Confirming these tables are not in use and ok to deprecate.

• EChetty moved this task from Incoming (new tickets) to Event Platform Backlog on the Data-Engineering board.Feb 10 2023, 12:34 PM

• EChetty moved this task from Event Platform Backlog to Incoming (new tickets) on the Data-Engineering board.

• EChetty edited projects, added Data-Engineering-Planning; removed Data-Engineering.Feb 10 2023, 12:38 PM

• EChetty moved this task from Backlog to Pipelines on the Data-Engineering-Planning board.Feb 10 2023, 12:40 PM

• EChetty added a project: Data Pipelines.

• EChetty moved this task from Backlog to To be discussed /To be estimated on the Data Pipelines board.Feb 10 2023, 12:49 PM

mpopov mentioned this in T327458: Document Traffic Datasets in Datahub.Feb 10 2023, 3:04 PM

Consider doing this at the same time as https://phabricator.wikimedia.org/T329978

JArguello-WMF set the point value for this task to 3.Mar 17 2023, 4:23 PM

JArguello-WMF moved this task from To be discussed /To be estimated to Next Up (revisit every 2 sprints) on the Data Pipelines board.

JAllemandou claimed this task.Mar 22 2023, 5:44 PM

• lbowmaker moved this task from Next Up (revisit every 2 sprints) to Sprint 11 on the Data Pipelines board.Mar 22 2023, 9:01 PM

• lbowmaker edited projects, added Data Pipelines (Sprint 11); removed Data Pipelines.

JArguello-WMF removed JAllemandou as the assignee of this task.Apr 3 2023, 4:09 PM

Consider implementing together with https://phabricator.wikimedia.org/T329978

JArguello-WMF edited projects, added Data Pipelines (Sprint 12); removed Data Pipelines (Sprint 11).Apr 24 2023, 1:16 PM

mforns claimed this task.Apr 25 2023, 7:11 PM

mforns moved this task from Next Up to In Progress on the Data Pipelines (Sprint 12) board.

Mentioned in SAL (#wikimedia-analytics) [2023-05-10T15:50:04Z] <mforns> killed oozie job mobile_apps-uniques-monthly-coord because of https://phabricator.wikimedia.org/T329310

Mentioned in SAL (#wikimedia-analytics) [2023-05-10T15:51:48Z] <mforns> stopped Airflow DAG mobile_app_session_metrics_weekly because of https://phabricator.wikimedia.org/T329310

This is the MR that will remove the DAG from airflow-dags:
https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/379

JArguello-WMF edited projects, added Data Pipelines (Sprint 14); removed Data Pipelines (Sprint 12).May 12 2023, 3:41 PM

JArguello-WMF moved this task from Next Up to In Progress on the Data Pipelines (Sprint 14) board.

mforns updated the task description. (Show Details)May 12 2023, 3:50 PM

Change 919357 had a related patch set uploaded (by Mforns; author: Mforns):

[analytics/refinery/source@master] Remove deprecated code for AppSessionMetrics

https://gerrit.wikimedia.org/r/919357

gerritbot added a project: Patch-For-Review.May 12 2023, 3:50 PM

mforns updated the task description. (Show Details)May 12 2023, 3:51 PM

mforns updated the task description. (Show Details)May 12 2023, 4:03 PM

Mentioned in SAL (#wikimedia-analytics) [2023-05-12T16:05:06Z] <mforns> dropped mobile_apps_* hive tables because of https://phabricator.wikimedia.org/T329310

mforns moved this task from In Progress to Blocked/Paused on the Data Pipelines (Sprint 14) board.May 12 2023, 4:05 PM

JArguello-WMF moved this task from Blocked/Paused to Next Up on the Data Pipelines (Sprint 14) board.Jun 5 2023, 4:08 PM

mforns moved this task from Next Up to In Progress on the Data Pipelines (Sprint 14) board.Jun 21 2023, 2:00 PM

mforns updated the task description. (Show Details)Jun 21 2023, 2:57 PM

Change 919357 merged by jenkins-bot:

[analytics/refinery/source@master] Remove deprecated code for AppSessionMetrics

https://gerrit.wikimedia.org/r/919357

Maintenance_bot removed a project: Patch-For-Review.Jun 21 2023, 3:31 PM

mforns updated the task description. (Show Details)Jun 21 2023, 3:31 PM

Change 931959 had a related patch set uploaded (by Mforns; author: Mforns):

[analytics/refinery@master] Remove queries for deprecated mobile_apps jobs

https://gerrit.wikimedia.org/r/931959

gerritbot added a project: Patch-For-Review.Jun 21 2023, 3:37 PM

mforns updated the task description. (Show Details)Jun 21 2023, 3:42 PM

mforns updated the task description. (Show Details)Jun 21 2023, 3:47 PM

mforns updated the task description. (Show Details)

Since after 30 days nobody has complained about the data missing:
Removed the refinery-source code and the airflow-dags code.
Also deleted the data from HDFS. It went to the Trash folder, where it will be irreversibly deleted in 30 days.
I think this can be moved to Done.

mforns moved this task from In Progress to Done on the Data Pipelines (Sprint 14) board.Jun 21 2023, 3:49 PM

JArguello-WMF closed this task as Resolved.Jun 29 2023, 4:29 PM

Change 931959 merged by Mforns:

[analytics/refinery@master] Remove queries for deprecated mobile_apps jobs

https://gerrit.wikimedia.org/r/931959

Maintenance_bot removed a project: Patch-For-Review.Sep 21 2023, 3:32 PM