Per T357537 and last week's discussion, we believe it is possible to alert on using the Prometheus metrics.
Creating this ticket to:
- Create Airflow alerts based on Prometheus metrics (see the PromQL queries in this dashboard for an idea of available metrics)
- Verify operation