Implement a data retention policy for webrequest_frontend datasets
Open, Needs TriagePublic
Actions

Assigned To

Authored By

	gmodena
	Nov 4 2024, 8:07 PM

Description

Raw, refined, data loss partitions should be automatically purged according to a data retention policy.

AC:

a retention policy is agreed upon and documented.
a retention policy is enforced for webrequest_frontend (haproxykafka) data.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		gmodena	T354694 [Maintenance] Safeguard VarnishKafka to HAProxy analytics transition
		Open		gmodena	T379024 Implement a data retention policy for webrequest_frontend datasets

Event Timeline

gmodena created this task.Nov 4 2024, 8:07 PM

gmodena mentioned this in T354694: [Maintenance] Safeguard VarnishKafka to HAProxy analytics transition.

gmodena moved this task from Next Up to In progress on the Data-Engineering (Q2 2024 October 1st - December 31th) board.Nov 25 2024, 3:15 PM

gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/940

wmf_airflow_common: enforce data retention.

gmodena mentioned this in T378342: [HAProxy transition] Deploy a staging airflow dag for webrequest refinement .Dec 2 2024, 11:45 AM

RFC for extending DataRegistry to support data retention policies for HiveDatasets: wmf_airflow_common Datasets retention policy

gmodena updated the task description. (Show Details)Dec 2 2024, 11:52 AM

gmodena updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/940

wmf_airflow_common: add drop_older_than utility method.

A lot of discussion about this task happened in slack / OTR. While we don't have a standard way to enforce data retention for HiveDatasets in airflow, we do have a couple of refinery scripts in place currently orchestrated via systemd timer. Search has a wrapper for these scripts, and runs them on Airflow via BashOperator.

I have two works streams, that want to generalize data retention policies in Airflow:

An RFC for extending DataRegistry to support data retention policies for HiveDatasets: wmf_airflow_common Datasets retention policy
An open MR to allow executing bash scripts on Airflow, which we need to use existing refinery business logic. If's a follow up form this slack thread.