Page MenuHomePhabricator
Create Task
Maniphest T411988

Airflow main performance instance optimization
Closed, ResolvedPublic
Actions

Description

After adding all our pipelines to it our main instance is now under pressure, especially from dynamic dags. This is especially visible from its database perspective (PG).

This ticket holds subtasks on this subject.

Related Objects
Search...

Event Timeline

Antoine_Quhen created this task.Dec 8 2025, 9:52 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 8 2025, 9:52 AM
Antoine_Quhen added subtasks: T392668: refine_to_hive dag optimizations, T375064: Move more of refine_hive_hourly dag logic into RefineConfiguration, T411989: Optimize canary event generation resources consumption on Airflow, T411990: Analyze and optimize Airflow Postgres backend performance, T411992: Reduce main Airflow DB size and consider splitting heavy workloads into separate instances.Dec 8 2025, 10:02 AM
BTullis added a subtask: T412003: Airflow-main scheduler loop sometimes slows down markedly.Dec 8 2025, 10:50 AM
Ahoelzl renamed this task from Airflow main instance optimization to Airflow main performance instance optimization.Dec 12 2025, 12:17 AM
Ahoelzl triaged this task as High priority.
Ahoelzl subscribed.
Ahoelzl changed the status of subtask T392668: refine_to_hive dag optimizations from In Progress to Open.Jan 12 2026, 9:27 PM
Antoine_Quhen closed subtask T375064: Move more of refine_hive_hourly dag logic into RefineConfiguration as Declined.Jan 12 2026, 11:04 PM
amastilovic subscribed.Jan 13 2026, 1:28 PM
JAllemandou mentioned this in T414460: Socket leaking on some dse-k8s row C & D hosts.Jan 13 2026, 5:32 PM
JAllemandou closed subtask T412003: Airflow-main scheduler loop sometimes slows down markedly as Resolved.Jan 19 2026, 12:43 PM
Ahoelzl moved this task from Incoming (new tickets) to Q3 FY25/26 January 1st - March 31th on the Data-Engineering board.Jan 28 2026, 5:06 PM
Ahoelzl edited projects, added: Data-Engineering (Q3 FY25/26 January 1st - March 31th); removed: Data-Engineering.
Antoine_Quhen closed subtask T411992: Reduce main Airflow DB size and consider splitting heavy workloads into separate instances as Resolved.Jan 28 2026, 7:01 PM
Ahoelzl closed subtask T411989: Optimize canary event generation resources consumption on Airflow as Resolved.Feb 2 2026, 6:30 PM
Antoine_Quhen reopened subtask T411992: Reduce main Airflow DB size and consider splitting heavy workloads into separate instances as Open.Feb 9 2026, 8:33 AM
Ahoelzl edited projects, added: Data-Engineering (Q4 FS25/26 April 1st - June 30st); removed: Data-Engineering (Q3 FY25/26 January 1st - March 31th).Apr 3 2026, 10:23 PM
Ahoelzl closed subtask T411992: Reduce main Airflow DB size and consider splitting heavy workloads into separate instances as Resolved.Apr 3 2026, 10:29 PM
Ahoelzl assigned this task to Antoine_Quhen.Jun 16 2026, 11:43 PM
Ahoelzl edited projects, added: Data-Engineering (Q1 FS26/27 July 1st - September 30th); removed: Data-Engineering (Q4 FS25/26 April 1st - June 30st).

@Antoine_Quhen assigning to you for tracking

Antoine_Quhen closed this task as Resolved.Tue, Aug 4, 7:35 AM
Antoine_Quhen closed subtask T411990: Analyze and optimize Airflow Postgres backend performance as Resolved.
Antoine_Quhen moved this task from Next Up to Done on the Data-Engineering (Q1 FS26/27 July 1st - September 30th) board.

Root cause was demanding xcoms network loads for airflow PG and location of PG instance on a machine with 1GB nic.
Now solved by pushing big xcoms on s3.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits