The refine_to_hive_hourly airflow job uses very large XCOM objects (see https://airflow.wikimedia.org/xcom/list/) for mapped-task configuration.
This creates a bottleneck on the DB networking side, leading to high scheduler-loop duration.
Multiple ideas of improvement:
- Improve the airflow code to do less calls to the configuration object
- Improve the airflow code to store smaller configuration object
- Store big XCOM objects on S3 instead of DB - https://airflow.apache.org/docs/apache-airflow/2.10.5/core-concepts/xcoms.html#object-storage-xcom-backend