Homogeneize maintenance DAGs across all airflow instances
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	brouberol
	Feb 10 2026, 1:40 PM

Project Tags

Referenced Files

	F72171564: image.png
	Feb 18 2026, 7:24 PM

	F72171561: image.png
	Feb 18 2026, 7:24 PM

	F72171548: image.png
	Feb 18 2026, 7:19 PM

Subscribers

Aklapper

brouberol

Tokens

"Pterodactyl" token, awarded by xcollazo.

Description

While working on T415777, I realized that some instances didn't have the log cleaner DAG running, and some didn't have any maintenance DAG deployed at all.

We need to make sure that all instances are managed the same way.

Details

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
test_k8s/external_task_sensor: update the monitored DAG id	repos/data-engineering/airflow-dags!2028	brouberol	T417011	main
log/xcom_cleaner: batch deletions by 1000 records	repos/data-engineering/airflow-dags!2016	brouberol	T417011	main
ml/log_cleaner: fix typo in import	repos/data-engineering/airflow-dags!2015	brouberol	T417011	main
Homogenenize maintenance dags across all instances	repos/data-engineering/airflow-dags!1999	brouberol	T417011	main

Customize query in GitLab

Related Objects

Mentioned In: T417867: Log maintenance DAG issues way too many S3 requests
Mentioned Here: T415777: Schedule the regular cleanup of large XCOM values from S3

Event Timeline

brouberol created this task.Feb 10 2026, 1:40 PM

brouberol triaged this task as Medium priority.

Herald added a subscriber: Aklapper. · View Herald TranscriptFeb 10 2026, 1:40 PM

brouberol changed the task status from Open to In Progress.Feb 10 2026, 1:40 PM

brouberol claimed this task.

brouberol moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.

brouberol edited projects, added: Data-Platform-SRE (2026.01.23 - 2026.02.13); removed: Data-Platform-SRE.

brouberol moved this task from In Progress to Needs Review on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.Feb 10 2026, 4:32 PM

Gehel edited projects, added: Data-Platform-SRE (2026-02-13 - 2026-03-06); removed: Data-Platform-SRE (2026.01.23 - 2026.02.13).Feb 16 2026, 3:33 PM

Gehel moved this task from Backlog - project to Needs Review on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1999

Homogenenize maintenance dags across all instances

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2015

ml/log_cleaner: fix typo in import

xcollazo awarded a token.Feb 18 2026, 3:12 PM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/2016

log/xcom_cleaner: batch deletions by 1000 records

All maintenance DAGs have been deployed, and I also had to refactor the log cleanup DAG to work in batches of 1000 files. As a result, we removed 20% of the overall files stored in S3 (old task logfiles > 6 months old)