Change Details

While doing opsweek, twice I found myself in a position that a Spark job was OOMing, yet the controlling Airflow job had no varprops (or DagProperties) for me to be able to easily tune the job. ( See [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/469 | here ]] and [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/471 | here ]] ) In this task we should: [] Identify which DAGs are missing the following Spark tunings: ``` driver_memory driver_cores executor_memory executor_cores ``` [] Modify the dags so that these tunings are available. Side note: It seems to me like the best way to do this would be as part of T336738, but leaving this as a separate ticket as this issue affects the opsweek sanity.

While doing opsweek, twice I found myself in a position that a Spark job was OOMing, yet the controlling Airflow job had no varprops (or DagProperties) for me to be able to easily tune the job. ( See [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/469 | here ]] and [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/471 | here ]] ) In this task we should: [] Identify which DAGs are missing the following Spark tunings: ``` driver_memory driver_cores executor_memory executor_cores ``` [] Modify the dags so that these tunings are available. Side note: It seems to me like the best time to do this would be as part of T336738, but leaving this as a separate ticket as this issue affects the opsweek sanity.

While doing opsweek, twice I found myself in a position that a Spark job was OOMing, yet the controlling Airflow job had no varprops (or DagProperties) for me to be able to easily tune the job. ( See [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/469 | here ]] and [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/471 | here ]] ) In this task we should: [] Identify which DAGs are missing the following Spark tunings: ``` driver_memory driver_cores executor_memory executor_cores ``` [] Modify the dags so that these tunings are available. Side note: It seems to me like the best waytime to do this would be as part of T336738, but leaving this as a separate ticket as this issue affects the opsweek sanity.