Page MenuHomePhabricator

Airflow dynamic task mapping logs mix up when, on rerun, an id is mapped to a different map_index_template
Open, Needs TriagePublic

Description

I got very confused when a job that had 'succeeded' previously suddenly started failing.

After investigating, I notice that the logs do not match because my operator that generates the dynamic tasks created extra jobs on a rerun, and thus the logs got mixed up.

Example log for try #1. This refers to map_index_template=minwiktionary.

Example log for try #3. This refers to map_index_template=minwikisource.

This seems to happen because we are indexing the logs by map_index=479, instead of by map_index_template=minwikisource.

I wonder if there is any way to index these logs by the map_index_template instead?

Event Timeline

Gehel subscribed.

Is this something that Data-Engineering can work on? At least investigate what kind of configuration or code change would be needed? We're also not sure what you mean by "indexing the logs".

We can try to implement the same pattern as for Refine fix proposed here: https://phabricator.wikimedia.org/T418151

Root cause:

This class of bug comes from how the dynamic tasks are generated: the configuration (the stream/wiki list) is fetched once, at the start of the DAG run, from ESC.

The dynamic task mapping then indexes tasks by their numerical map_index (0, 1, 2, …), which is just a positional pointer into that list. If the list changes between runs (ex: an entry added, removed, or reordered) the same map_index now points to a different stream than it did on the original run. That's why the logs appear "mixed up", because the underlying list shifted.

Possible fix:

The proper fix would be to snapshot the config somewhere stable (xcom ?) (so a given run always resolves the same map_index), rather than re-fetching it from ESC on each attempt. But it introduces its own consistency problems (sometimes config update is expected) and adds complexity.

Given that, I'm unassigning myself for now.
Happy to pick it back up when the need to implement it becomes more pressing.