Maniphest T415777

Schedule the regular cleanup of large XCOM values from S3
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	brouberol
	Jan 28 2026, 12:35 PM

Project Tags

Referenced Files

None

Subscribers

Description

In T415661, we configured Airflow to store XCOM values larger than 256KB into S3. We need to take care of the garbage collection of these values, after a delay of, say, 7 days.

Details

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Deploy the XCOM cleaner DAG to all instances	repos/data-engineering/airflow-dags!1997	brouberol	T415777	main
	Add a DAG factory to cleanup old XCOMs from S3	repos/data-engineering/airflow-dags!1986	brouberol	T415777	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Antoine_Quhen	T411988 Airflow main performance instance optimization
		Resolved	BUG REPORT	JAllemandou	T412003 Airflow-main scheduler loop sometimes slows down markedly
		Resolved		brouberol	T415661 Automatically put large Airflow XCOM values in S3
		Resolved		brouberol	T415777 Schedule the regular cleanup of large XCOM values from S3

Event Timeline

brouberol created this task.Jan 28 2026, 12:35 PM

brouberol triaged this task as Medium priority.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 28 2026, 12:35 PM

Gehel edited projects, added: Data-Platform-SRE (2026.01.23 - 2026.02.13); removed: Data-Platform-SRE.Jan 30 2026, 10:24 AM

Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.

brouberol changed the task status from Open to In Progress.Feb 9 2026, 1:17 PM

brouberol claimed this task.

brouberol moved this task from Backlog - operations to In Progress on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.

brouberol moved this task from In Progress to Needs Review on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.

I've opened a patch (https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1986) that cleans up XCOMs that are > 30d

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1986

Add a DAG factory to cleanup old XCOMs from S3

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1997

Deploy the XCOM cleaner DAG to all instances

The DAG has been deployed and unpaused into all instances.

brouberol closed this task as Resolved.Feb 10 2026, 1:39 PM

brouberol moved this task from Needs Review to Done on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.

brouberol mentioned this in T417011: Homogeneize maintenance DAGs across all airflow instances.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.Feb 13 2026, 8:42 AM