While investigating a task failure in Mjolnir (2025-01-15), we found that the DAG references a generic refinery-hive-shaded.jar artifact.
If the artifact is refreshed (e.g., re-deployed by third parties) while a Spark job is running, executors will start to fail.
AC:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| search: mjolnir: pin refinery-hive version | repos/data-engineering/airflow-dags!1026 | gmodena | mjolnir-pin-refinery | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T383046 [Epic] Improve Machine Learning Ranking models for Search | |||
| Resolved | gmodena | T383048 Investigate current MLR models for Search and identify improvements | |||
| Resolved | gmodena | T383870 mjolnir should pin refinery jar version explicitly |
gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1026
search: mjolnir: pin refinery-hive version
gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1026
search: mjolnir: pin refinery-hive version