Approved in 2021-02-03 WMCS team meeting. @Bstorm will also provide some information on tricks for scheduling large pods in a comment here.

I just checked your pod definitions and see that you are setting requests equal to limits, which is great. The scheduler will block off the requests on the node, so if you know it is going to be a heavy RAM user, you should set request to what you actually expect it to use in an average run.

I think you need a limitrange increase more than namespace quota, which is not reflected in the document (updating that now) so that each container can have more than 4GB of RAM. I see your web service is using default resources, so you would likely still have 7.5 GB for the namespace to consume per job. The OOM killer is affected by the resource limits rather than the resource quotas. The quotas will block the launch of a pod that requests more than the total the namespace is allowed. On CPU, I could do the same thing of increasing your limitrange to 2 per container, and then if you launch requesting 1.75, you'd be within quota, or I could expand that to 3 so you could use a full 2 for the job containers.

So! All that said, my current recommendation would be to expand the limitrange for your namespace to 8Gi RAM and the resourcequota to something like 9Gi to account for the web service as well? Does that seem reasonable.

The equivalent command (now in the doc) is:

tools.oabot@tools-sgebastion-08:~$ kubectl describe limitranges
Name:       tool-oabot
Namespace:  tool-oabot
Type        Resource  Min    Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---    ---  ---------------  -------------  -----------------------
Container   cpu       50m    1    150m             500m           -
Container   memory    100Mi  4Gi  256Mi            512Mi          -

Does that seem reasonable.

Sure, sounds good, let's try. Thank you!

I tried surfing closer to the hard limits at the beginning, but I got annoying errors like (P13646):

Warning  FailedCreate  7m14s  job-controller  Error creating: pods "test-pqt5d" is forbidden: [maximum cpu usage per Container is 1, but limit is 2, maximum memory usage per Container is 4Gi, but limit is 7680Mi]

So that's indeed the limitranges.

Ok, so I've bumped the resource quota a bit to accomodate the new limitrange:

Name:                   tool-oabot
Namespace:              tool-oabot
Resource                Used    Hard
--------                ----    ----
configmaps              1       10
limits.cpu              2       3
limits.memory           4608Mi  9Gi
persistentvolumeclaims  0       3
pods                    2       4
replicationcontrollers  0       1
requests.cpu            1500m   2
requests.memory         4352Mi  9Gi
secrets                 1       10
services                1       1
services.nodeports      0       0

And I bumped up the limit range to a level that might work.

Name:       tool-oabot
Namespace:  tool-oabot
Type        Resource  Min    Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---    ---  ---------------  -------------  -----------------------
Container   cpu       50m    2    150m             500m           -
Container   memory    100Mi  8Gi  256Mi            512Mi          -

You should now be able to launch a pod with a CPU request of 2 and a RAM of 8 GB.