I had the same thing happening again now when I tried starting another job. This one started after about half an hour, I think. It also has an event talking about "Insufficient memory":

Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  3m18s (x7 over 6m6s)  default-scheduler  0/81 nodes are available: 1 Insufficient memory, 11 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3 node(s) had untolerated taint {toolforge.org/gateway: true}, 64 Insufficient cpu. preemption: 0/81 nodes are available: 17 Preemption is not helpful for scheduling, 64 No preemption victims found for incoming pod.
  Warning  FailedScheduling  58s (x26 over 5m57s)  default-scheduler  0/81 nodes are available: 11 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3 node(s) had untolerated taint {toolforge.org/gateway: true}, 64 Insufficient cpu. preemption: 0/81 nodes are available: 17 Preemption is not helpful for scheduling, 64 No preemption victims found for incoming pod.

The whole log (relevant job at the end, named "make-declarations-wlm-2017-vc4s4"):

Thank you for the detailed report @Sebastian_Berlin-WMSE !

I checked commonsdb-registry usage patterns here https://grafana.wmcloud.org/goto/dfn7icfiitcsgb?orgId=1 and it seems actual cpu usage is far lower than cpu requested across the whole namespace/tool; does that check out ? if that's the case then as an immediate bandaid to get your pods scheduled faster is to lower per-job cpu requests to 0.5 for example.

I'll also followup to get more kube scheduler metrics collected

That's quite possible. I haven't really checked CPU usage. I was just hoping more CPUs means more faster 🙂 I'd guess that the CPU usage fluctuates for the different steps in the process that the tool runs.

Is it possible for me to view the page that you linked to or see it somewhere else? I get "Unauthorized". It would help if I want to experiment with the resources.

In T427204#11954411, @Sebastian_Berlin-WMSE wrote:

That's quite possible. I haven't really checked CPU usage. I was just hoping more CPUs means more faster 🙂 I'd guess that the CPU usage fluctuates for the different steps in the process that the tool runs.

Unfortunately not that simple ! As you'll see in the link below, the sum of CPU across the whole k8s namespace (i.e. all pods) does not go over 0.5 cpu. Requesting 0.3 - 0.5 cpu per pod/job should be enough and make scheduling easier. Something else to keep in mind is that those are cpu you are requesting to get your pod to start, the actual cpu limit where your tool will get throttled is and can be higher.

Is it possible for me to view the page that you linked to or see it somewhere else? I get "Unauthorized". It would help if I want to experiment with the resources.

My bad! The dashboard is public though I linked the grafana-rw endpoint which requires authentication. Here's the panel I was referring to https://grafana.wmcloud.org/d/TJuKfnt4z/tool-dashboard?orgId=1&from=now-30d&to=now&timezone=utc&var-cluster=P8433460076D33992&var-namespace=tool-commonsdb-registry&viewPanel=panel-1

Looks like you linked to the "secret" panel again 🙂 I took a shot at removing the "-rw" at the start and that worked.

You're right, looks like the actual CPU usage is much less than what I expected. I'll keep that in mind for future jobs.