Maniphest T411988

Airflow main performance instance optimization
Open, HighPublic
Actions

Assigned To

None

Authored By

	Antoine_Quhen
	Dec 8 2025, 9:52 AM

Tags

Referenced Files

None

Subscribers

Description

After adding all our pipelines to it our main instance is now under pressure, especially from dynamic dags. This is especially visible from its database perspective (PG).

This ticket holds subtasks on this subject.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		None	T307500 Airflow Hackathon (May 2022)
		Open		Ahoelzl	T307505 Refine jobs should be scheduled by Airflow
		Resolved		Antoine_Quhen	T369845 [Refine Refactoring] Refine jobs should be scheduled by Airflow: deployment
		Open		None	T411988 Airflow main performance instance optimization
		Open		None	T392668 refine_to_hive dag optimizations
		Declined		Antoine_Quhen	T375064 Move more of refine_hive_hourly dag logic into RefineConfiguration
		Resolved		Antoine_Quhen	T411989 Optimize canary event generation resources consumption on Airflow
		Open		None	T411990 Analyze and optimize Airflow Postgres backend performance
		Open		Antoine_Quhen	T411992 Reduce main Airflow DB size and consider splitting heavy workloads into separate instances
		Resolved	BUG REPORT	JAllemandou	T412003 Airflow-main scheduler loop sometimes slows down markedly
		Resolved		Antoine_Quhen	T414810 Switch Refine DAG priority weights to absolute
		Open		None	T414953 Refine generates very large XCOM values
		Resolved		brouberol	T415635 Force postgresql-airflow-main pods to run on kubernetes workers equipped with a 10G NIC
		Resolved		brouberol	T415661 Automatically put large Airflow XCOM values in S3
		Resolved		brouberol	T415777 Schedule the regular cleanup of large XCOM values from S3
		Resolved		brouberol	T416821 Implement a way to fetch the content of an XCOM when stored in S3

Event Timeline

Antoine_Quhen created this task.Dec 8 2025, 9:52 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 8 2025, 9:52 AM

Antoine_Quhen added subtasks: T392668: refine_to_hive dag optimizations, T375064: Move more of refine_hive_hourly dag logic into RefineConfiguration, T411989: Optimize canary event generation resources consumption on Airflow, T411990: Analyze and optimize Airflow Postgres backend performance, T411992: Reduce main Airflow DB size and consider splitting heavy workloads into separate instances.Dec 8 2025, 10:02 AM

BTullis added a subtask: T412003: Airflow-main scheduler loop sometimes slows down markedly.Dec 8 2025, 10:50 AM

Ahoelzl renamed this task from Airflow main instance optimization to Airflow main performance instance optimization.Dec 12 2025, 12:17 AM

Ahoelzl triaged this task as High priority.

Ahoelzl subscribed.

Ahoelzl changed the status of subtask T392668: refine_to_hive dag optimizations from In Progress to Open.Mon, Jan 12, 9:27 PM

Antoine_Quhen closed subtask T375064: Move more of refine_hive_hourly dag logic into RefineConfiguration as Declined.Mon, Jan 12, 11:04 PM

amastilovic subscribed.Tue, Jan 13, 1:28 PM

JAllemandou mentioned this in T414460: Socket leaking on some dse-k8s row C & D hosts.Tue, Jan 13, 5:32 PM

JAllemandou closed subtask T412003: Airflow-main scheduler loop sometimes slows down markedly as Resolved.Mon, Jan 19, 12:43 PM

Ahoelzl moved this task from Incoming (new tickets) to Q3 FY25/26 January 1st - March 31th on the Data-Engineering board.Wed, Jan 28, 5:06 PM

Ahoelzl edited projects, added Data-Engineering (Q3 FY25/26 January 1st - March 31th); removed Data-Engineering.

Antoine_Quhen closed subtask T411992: Reduce main Airflow DB size and consider splitting heavy workloads into separate instances as Resolved.Wed, Jan 28, 7:01 PM

Ahoelzl closed subtask T411989: Optimize canary event generation resources consumption on Airflow as Resolved.Mon, Feb 2, 6:30 PM

Antoine_Quhen reopened subtask T411992: Reduce main Airflow DB size and consider splitting heavy workloads into separate instances as Open.Mon, Feb 9, 8:33 AM