Trying to spin up a spark job with --executor-cores greater than 4 is able to start, but it is never assigned any executors from yarn. This looks to be limited by the configuration key yarn.scheduler.maximum-allocation-cores:
ebernhardson@stat1005:~$ hdfs getconf -confKey yarn.scheduler.maximum-allocation-vcores Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 4
I'd like to experiment with different values to figure out what the most efficient use of resources is when training ML models. It may be that fewer executors with more cores per executor is more efficient (or it might not) in terms of total cpu time used. To find out i would need to be able to test,