Page MenuHomePhabricator
Create Task
Maniphest T357430

Airflow mapped tasks UI & metrics
Closed, ResolvedPublic13 Estimated Story Points
Actions

Description

The current caveats of the dynamic DAG in Airflow for us are the UI & the metrics.

1/ UI

Currently, in Airflow 2.8.1, The UI references the map tasks with integer indexes.

A pull request has been opened to begin implementing a custom templated index that will replace the number in the UI.
https://github.com/apache/airflow/pull/36797

At the time of writing, the use of the new parameter in the UI is not implemented.

2/ Metrics

The last version of Airflow sends metrics about mapped tasks without providing the index or the custom index.

It would be nice to include them as a tag to follow the different tasks through Grafana. We could contribute to the previous PR.

Related Objects
Search...

Event Timeline

Antoine_Quhen created this task.Feb 13 2024, 3:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 13 2024, 3:39 PM
Antoine_Quhen added a parent task: T356360: [Refine Refactoring] Orchestrate Airflow execution of navigationtiming from config store.Feb 13 2024, 3:39 PM
• lbowmaker moved this task from Incoming (new tickets) to Backlog on the Data-Engineering board.Feb 13 2024, 9:54 PM
• lbowmaker set the point value for this task to 13.Feb 16 2024, 7:53 PM
• lbowmaker moved this task from Backlog to Sprint 9 on the Data-Engineering board.Feb 19 2024, 3:18 PM
• lbowmaker edited projects, added Data-Engineering (Sprint 9); removed Data-Engineering.
Ahoelzl assigned this task to Antoine_Quhen.Mar 21 2024, 2:00 AM
Ahoelzl moved this task from Next Up to In progress on the Data-Engineering (Sprint 9) board.
Antoine_Quhen added a comment.Mar 26 2024, 4:15 PM

The Airflow PR has been merged and should be released in Airflow 2.9 in April.

• lbowmaker edited projects, added Data-Engineering (Q4 2024 April 1st - June 30th); removed Data-Engineering (Sprint 9).Apr 1 2024, 12:05 PM
• lbowmaker moved this task from Next Up to In progress on the Data-Engineering (Q4 2024 April 1st - June 30th) board.
• lbowmaker edited projects, added Data-Engineering (Q1 2024 July 1st - September 30th); removed Data-Engineering (Q4 2024 April 1st - June 30th).Jul 15 2024, 2:24 PM
• lbowmaker moved this task from Next Up to In progress on the Data-Engineering (Q1 2024 July 1st - September 30th) board.
Ottomata added a parent task: T365449: Upgrade Airflow to 2.9.3.Jul 18 2024, 4:15 PM
Ottomata moved this task from In progress to Blocked/Paused on the Data-Engineering (Q1 2024 July 1st - September 30th) board.Jul 18 2024, 4:41 PM
Ottomata added subscribers: Ahoelzl, Snwachukwu.Aug 14 2024, 9:06 PM
Ottomata subscribed.

Airflow 2.9 is deployed. Xabriel is naming mapped tasks here: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/804/diffs

Can we do this for canary_events dag asap?

:D

Ahoelzl moved this task from Blocked/Paused to In progress on the Data-Engineering (Q1 2024 July 1st - September 30th) board.Aug 15 2024, 4:56 PM
amastilovic subscribed.Sep 16 2024, 10:36 PM
amastilovic moved this task from In progress to Blocked/Paused on the Data-Engineering (Q1 2024 July 1st - September 30th) board.Sep 23 2024, 2:30 PM
Ahoelzl edited projects, added Data-Engineering (Q2 2024 October 1st - December 31th); removed Data-Engineering (Q1 2024 July 1st - September 30th).Oct 9 2024, 1:53 PM
Ahoelzl moved this task from Next Up to Blocked/Paused on the Data-Engineering (Q2 2024 October 1st - December 31th) board.
Ahoelzl edited projects, added Data-Engineering (Q3 2025 January 1st - March 31th); removed Data-Engineering (Q2 2024 October 1st - December 31th).Jan 18 2025, 12:31 AM
Ahoelzl moved this task from Next Up to Blocked/Paused on the Data-Engineering (Q3 2025 January 1st - March 31th) board.
Ahoelzl added a comment.Feb 20 2025, 2:57 PM

Moving this back to backlog.

Ahoelzl edited projects, added Data-Engineering; removed Data-Engineering (Q3 2025 January 1st - March 31th).Feb 20 2025, 2:58 PM
Ottomata moved this task from Incoming (new tickets) to Next Up on the Data-Engineering board.Mar 3 2025, 9:30 PM
Antoine_Quhen closed this task as Resolved.Apr 24 2025, 11:58 AM

Closing the ticket:
• UI: The improvements have already been implemented with the upgrade to Airflow 2.5.
• Metrics: The legacy Grafana dashboard for Airflow task metrics no longer shows data since the migration to Kubernetes. This has been partially replaced by a new Grafana dashboard for Airflow (Data Platform/Airflow Instance), along with enhancements in the Airflow UI itself, which now provides a cluster overview and basic task duration graphs.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits