Change Details

Refine currently generates a high overhead for relatively small work units: * Many templated calls generating individual requests/jobs. * Small streams processed as separate tasks instead of being grouped. * Precomputation of all configuration in configuration step and storage in xcoms This leads to: * Large numbers of task instances (200*3=600 tasks per run). * pressure on DB (large rows + number of rows) * Increased dependency check times. First improvements: * Run small jobs within pod (local spark) as it's done in analytics-test * Only store in xcoms the minimal information needed to launch downstream tasks T375064 * Compute configuration at execution time instead of precomputing all in configuration step We’ll start by testing medium and low resource profiles and adjust based on observations from: • [[ https://grafana.wikimedia.org/d/pz5A-vASz/kubernetes-resources | Kubernetes Resources dashboard ]] • [[ https://grafana.wikimedia.org/d/lFXqBGxWk/airflow-instance | Airflow Instance dashboard ]] • Pod-level stats Eventually, later, we can consider batching. We would loose some visibility on individual items but we would reduce DB pressure and number of task instances. Huge thanks to @brouberol for the guidance!

Refine currently generates a high overhead for relatively small work units: * Many templated calls generating individual requests/jobs. * Small streams processed as separate tasks instead of being grouped. * Precomputation of all configuration in configuration step and storage in xcoms This leads to: * Large numbers of task instances (200*3=600 tasks per run). * pressure on DB (large rows + number of rows) * Increased dependency check times. First improvements: * Run small jobs within pod (local spark) as it's done in analytics-test * Only store in xcoms the minimal information needed to launch downstream tasks T375064 * Compute configuration at execution time instead of precomputing all in configuration step We’ll start by testing medium and low resource profiles and adjust based on observations from: • [[ https://grafana.wikimedia.org/d/pz5A-vASz/kubernetes-resources | Kubernetes Resources dashboard ]] • [[ https://grafana.wikimedia.org/d/lFXqBGxWk/airflow-instance | Airflow Instance dashboard ]] • Pod-level stats Eventually, later, we can consider batching. We would lose some visibility on individual items but we would reduce DB pressure and number of task instances. Huge thanks to @brouberol for the guidance!

Refine currently generates a high overhead for relatively small work units: * Many templated calls generating individual requests/jobs. * Small streams processed as separate tasks instead of being grouped. * Precomputation of all configuration in configuration step and storage in xcoms This leads to: * Large numbers of task instances (200*3=600 tasks per run). * pressure on DB (large rows + number of rows) * Increased dependency check times. First improvements: * Run small jobs within pod (local spark) as it's done in analytics-test * Only store in xcoms the minimal information needed to launch downstream tasks T375064 * Compute configuration at execution time instead of precomputing all in configuration step We’ll start by testing medium and low resource profiles and adjust based on observations from: • [[ https://grafana.wikimedia.org/d/pz5A-vASz/kubernetes-resources | Kubernetes Resources dashboard ]] • [[ https://grafana.wikimedia.org/d/lFXqBGxWk/airflow-instance | Airflow Instance dashboard ]] • Pod-level stats Eventually, later, we can consider batching. We would loose some visibility on individual items but we would reduce DB pressure and number of task instances. Huge thanks to @brouberol for the guidance!