Page MenuHomePhabricator

Refine generates very large XCOM values
Open, Needs TriagePublic

Description

The refine_to_hive_hourly airflow job uses very large XCOM objects (see https://airflow.wikimedia.org/xcom/list/) for mapped-task configuration.
This creates a bottleneck on the DB networking side, leading to high scheduler-loop duration.
Multiple ideas of improvement:

Event Timeline

  • Option 1:
    • downstreaming xcoms (keeping only the necessary fields)
    • computing parameters for each task in a pre_execution function (1 for each task)

      This solution do not optimize xcom as much as we could. But it's a safe bet.
  • Option 2:
    • reducing xcom to stream name (+Refine tshirt sizes)
    • calling EventStreamConfig in pre_execution to fetch the config

      This solution would generate many calls to ESC.
  • Option 3:
    • Computing all tasks parameters within the configuration fetch task at beginning of the dag run (the task fetching the list from ESC). It will expand the tasks and set the parameters, saving them in the DB. No parameter computation will have to be made later.

      Xcom will be as small as possible. But we need to validate the idea with a POC.

Thanks @aqu for the summary above.
My 2 cents:
I would be ok to time-bound-try option 3, as if we manage to make it work it means great learning about airflow internals, and better ways to tweak it.
If we don't manage to make it work with option 3 in the defined amount of time, then option 1 would be my choice.

Just FYI, we're mitigating the effect of these large XCOM values in T415635 and T415661, which should give you the time you need to experiment and address the root cause the way you prefer.

Ahoelzl subscribed.

Given the progress on the Airflow infrastructure (leveraging s3 for large payloads) which seems to improve the issue, we are postponing the Refine refactoring for now.