Refine currently generates a high overhead for relatively small work units:
- Many templated calls generating individual requests/jobs.
- Small streams processed as separate tasks instead of being grouped.
- Precomputation of all configuration in configuration step and storage in xcoms
This leads to:
- Large numbers of task instances (200*3=600 tasks per run).
- pressure on DB (large rows + number of rows)
- Increased dependency check times.
First improvements:
- Run small jobs within pod (local spark) as it's done in analytics-test
- Only store in xcoms the minimal information needed to launch downstream tasks T375064
- Compute configuration at execution time instead of precomputing all in configuration step
We’ll start by testing medium and low resource profiles and adjust based on observations from:
• Kubernetes Resources dashboard
• Airflow Instance dashboard
• Pod-level stats
Eventually, later, we can consider batching. We would lose some visibility on individual items but
we would reduce DB pressure and number of task instances.
Huge thanks to @brouberol for the guidance!