We use kubernetes limitranges to limit how much resources single pods can use to prevent overloads of single nodes. By default they look like this:
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- Container cpu 50m 1 150m 500m - Container memory 100Mi 4Gi 256Mi 512Mi -
The jobs framework should reject jobs that try to use more than those amounts (which are configurable per-namespace), given that otherwise those job objects will just fail to create the pod in any case.