A number of airflow had failed tasks, but we had received no error email. It seems that all the failed tasks were all sensors.
This could lead to a critical failure going unnoticed
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | mforns | T311976 Investigate why airflow sensor tasks fail without sending errors | |||
| Resolved | xcollazo | T313734 Try to reproduce the silently failing airflow sensor error. | |||
| Resolved | Antoine_Quhen | T309552 Update Airflow DAGs code to make it compatible with version V2.3.4 of Airflow | |||
| Resolved | Antoine_Quhen | T325204 Add missing tests in airflow-dags |
Possible solutions:
Since we deleted some airflow logs under an-launcher1002:/srv/analytics-airflow/logs this issue has not happened.
Also, every time there was a sensor silent failure, at least one of the sensor's log files was missing (Airflow couldn't find it).
From this, and some team conversations, we suspect that the silent failures could be caused by the logs filling up the /srv disk in an-launcher1002.
So, I was looking at the logs and found out that an Airflow bug is polluting the logs (and breaking SLAs), see: https://phabricator.wikimedia.org/T314181#8116392
Things we can do:
The problem has happened again today.
I'm in favor of creating a dedicated alarm for airflow failed tasks: if any dagRun of an instance has a failed task, the alarm fires.
This would be feasible using the airflow rest API:
curl http://an-launcher1002.eqiad.wmnet:8600/api/v1/dags/~/dagRuns?state=failed
But, the state filter has been added in version 2.3.0 of airflow.
We therefore need to upgrade before having this - ping T315580 !