As part of the investigation in T412003, we're tracking down why the airflow-main scheduler repeatedly shows an increase in scheduler loop time. We recently caused all kubernetes nodes to rolling reboot, and as a result, these frequent latency peaks have subsided.
Previously to this reboot, the master postgresqlq pod was located on a Kubernetes node equipped with a 1G NIC, and after the rolling-reboot, it ended up (by chance) on a host equipped with a 10G card.
I believe the root cause is still the one explained in T414953, aka very large XCOMs being stored into and fetched from the database at a large scale, causing peaks of traffic between the scheduler and the database. When the master PG pod runs on a host with a 1G card, that traffic saturates the interface, and causes a peak of latency, probably due to TCP retries. When that pod is on a host equipped with a 10G NIC, then the network does not saturate.
Until we can fix the root cause and have the large XCOMs stored into and fetched from S3 (and/or until all our nodes are equipped with 10G NICs, cf T414787 )we should ensure that postgresql-airflow-main pods run on nodes equipped with 10G NICs, to mitigate the symptoms.