Similar idea than in T389771: we can link the airflow task instance page with related data. This time, the idea is to have a link between a specific dump v1 task instance and the dashboard showing the associated resource usage of the tas pod.
Description
Description
Details
Details
Related Changes in GitLab:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Display a link to grafana to graph the pod resources for all pods started with... | repos/data-engineering/airflow-dags!1204 | brouberol | T389927 | main | |
| test_k8s: Display a link to the grafana pod resources on a dumps v1 TI page | repos/data-engineering/airflow-dags!1182 | brouberol | T389927 | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | brouberol | T398436 Future improvements to the Dumps on Airflow configuration - tracking task | |||
| Open | None | T389927 Display a link to the Grafana pod usage dashboard in the airflow dump task instance page |
Event Timeline
Comment Actions
Preleminary tests seem to fail. No extra link is shown on the mapped task page. After a bit of searching, I found https://github.com/apache/airflow/issues/43757, that seem to be fixed by https://github.com/apache/airflow/pull/46337.
I'll need to upgrade test-k8s to airflow 2.10.5 to see if that fixes the issue.
Comment Actions
brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1182
test_k8s: Display a link to the grafana pod resources on a dumps v1 TI page
Comment Actions
Actually, I was able to use operator extra links with mapped tasks on airflow 2.10.5, now that https://github.com/apache/airflow/issues/43757 was integrated. We should be able to make it work post upgrade.
Comment Actions
brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow/-/merge_requests/51
Build an airflow 2.10.5 image
Comment Actions
I _think_ I've found yet another extra link / mapped task related bug: https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1743578980922949
I managed to make things work with non mapped tasks, which still might be useful for anyone using our WMFKubernetesPodOperator class.
Comment Actions
brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1204
Display a link to grafana to graph the pod resources for all pods started with...
Comment Actions
It seems like this has stopped working again.
The extra_links argument is present on the tasks, but it isn't shown.
I'll reopen the task because it would be useful to have it working, but it's not a blocker.