We don't really have any good way to tell why a Java OOM happened during a particular Spark job, other than checking stack traces.
In Spark 3.0 there is a direct support for Prometheus, but since we have 2.4.4 we might need to use something less sophisticated. One possible way could be to use https://spark.apache.org/docs/2.4.4/monitoring.html#metrics to log metrics in the Spark tasks' stdout, adding some config to spark-defaults. I briefly tried it in Hadoop test, getting something like:
LogType:stdout
Log Upload Time:Fri Jan 08 08:06:05 +0000 2021
LogLength:11698
Log Contents:
1/8/21 8:05:57 AM ==============================================================
-- Gauges ----------------------------------------------------------------------
application_1608216118485_10852.25.ExternalShuffle.shuffle-client.usedDirectMemory
value = 16777216
application_1608216118485_10852.25.ExternalShuffle.shuffle-client.usedHeapMemory
value = 16777216
application_1608216118485_10852.25.executor.filesystem.file.largeRead_ops
value = 0
application_1608216118485_10852.25.executor.filesystem.file.read_bytes
value = 0
application_1608216118485_10852.25.executor.filesystem.file.read_ops
value = 0
application_1608216118485_10852.25.executor.filesystem.file.write_bytes
value = 0
application_1608216118485_10852.25.executor.filesystem.file.write_ops
value = 0
application_1608216118485_10852.25.executor.filesystem.hdfs.largeRead_ops
value = 0
application_1608216118485_10852.25.executor.filesystem.hdfs.read_bytes
value = 0
application_1608216118485_10852.25.executor.filesystem.hdfs.read_ops
value = 0
application_1608216118485_10852.25.executor.filesystem.hdfs.write_bytes
value = 0
application_1608216118485_10852.25.executor.filesystem.hdfs.write_ops
value = 0
application_1608216118485_10852.25.executor.jvmCpuTime
value = 7630000000
application_1608216118485_10852.25.executor.threadpool.activeTasks
value = 0
application_1608216118485_10852.25.executor.threadpool.completeTasks
value = 0
application_1608216118485_10852.25.executor.threadpool.currentPool_size
value = 0
application_1608216118485_10852.25.executor.threadpool.maxPool_size
value = 2147483647
-- Counters --------------------------------------------------------------------
application_1608216118485_10852.25.HiveExternalCatalog.fileCacheHits
count = 0
application_1608216118485_10852.25.HiveExternalCatalog.filesDiscovered
count = 0
application_1608216118485_10852.25.HiveExternalCatalog.hiveClientCalls
count = 0
application_1608216118485_10852.25.HiveExternalCatalog.parallelListingJobCount
count = 0
application_1608216118485_10852.25.HiveExternalCatalog.partitionsFetched
...
...It can be very spammy of course but we could start enabling some metrics and see if they help or not.