Page MenuHomePhabricator

Analyze and optimize Airflow Postgres backend performance
Closed, ResolvedPublic

Description

The Airflow backend Postgres instance shows signs of performance pressure, particularly during busy periods (e.g. beginning of the month on the main instance). We need a systematic investigation of where time is spent and what the main bottlenecks are.

  • Identify the top offending queries that consume most of the DB resources (e.g. with pg_stat_statements, or slow-query logs). Leverage existing Airflow Postgres dashboards.
  • Determine whether the main bottleneck is CPU, IO, locks, external back pressure (Ceph), or N+1 patterns between core Airflow tables. Focus on beginning of the month load patterns.
  • Propose and implement targeted optimizations (new indexes, query changes, config tuning).

Event Timeline

Change #1218709 had a related patch set uploaded (by Aqu; author: Aqu):

[operations/deployment-charts@master] postgresql-airflow-main: Increase pgbouncer pool size

https://gerrit.wikimedia.org/r/1218709

Change #1218709 merged by jenkins-bot:

[operations/deployment-charts@master] postgresql-airflow-main: Increase pgbouncer pool size

https://gerrit.wikimedia.org/r/1218709

Root cause was demanding xcoms network loads for airflow PG and location of PG instance on a machine with 1GB nic.
Now solved by pushing big xcoms on s3.