[23:18]  <   danilo> I got the following mensage when using 'kubectl describe pod ...' for wmopbot: "Events: ... Warning  FailedScheduling  4m25s (x70 over 23m)  default-scheduler  0/64 nodes are available ..."
[23:27]  <    bd808> danilo: There was a ceph storage blip that messed up NFS so that the kubernetes nodes needed to be restarted. I think what you are seeing there was a side effect of the restart script that t.aavi ran to do the restarts. I was just able to schedule a new pod and `kubectl get node` is now showing all nodes as "Ready".
[23:30] bd808 sees that pod is still pending and looks closer
[23:31]  <    bd808> Looks like the problem is "58 Insufficient cpu." How many cores is that pod trying to get?
[23:33]  <    bd808> Looks like the default limits. This isn't making sense to me yet. The pod is asking for cpu:250m which is tiny.
[23:36]  <    bd808> danilo: I am going to delete that pending pod and see if anything different happens when the ReplicaSet tries to recreate it.
[23:37]  <    bd808> huh. same issue.
[23:37]  <   danilo> ok, feel free to delete, restart or do anythink that can help to solve the problem
[23:39]  <    bd808> "0/64 nodes are available: 3 node(s) had taint {ingressgen2: true}, that the pod didn't tolerate, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 58 Insufficient cpu." is the FailedScheduling description for those following along at home.
[23:41]  <    taavi> bd808: hm. I wonder if we're genuinely running out of capacity with all of the grid migrations going on
[23:42]  <    bd808> taavi: it's possible I guess. There are 20-40 pods running on each worker node right now
[23:42]  <    taavi> uh wat. https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1&forceLogin&from=now-2d&to=now&viewPanel=36
[23:43]  <    taavi> that's a very rapid increase in the total CPU requests starting at around 21 UTC