Recently, we encountered occasional memory-related failures in mjolnir that require investigation. It is possible that the Spark configuration can be tuned to better utilize YARN resources under the current workloads.
AC
gmodena updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1032
search: mjolnir: bump driver memory.
gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1032
search: mjolnir: bump driver memory.
gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1034
search: mjonlnir: set skein memory for feature selection
gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1034
search: mjonlnir: set skein memory for feature selection
After switching Spark deployment from cluster to client mode, we encountered memory pressure issues where the Spark driver was under-resourced.
This occurred because the driver was running in a Skein-managed container instead of directly atop YARN, and skein was improperly handling resource management.
This issue was resolved by: