So the CNI path there is wrong, and our containerd config Puppetization is supposed to change that. That node was affected by T358179: [wmcs-cookbooks] wmcs.toolforge.add_k8s_node occasionally fails to setup custom Puppetmaster, so I think the reason why your pod was affected was that I failed to drain + reboot that node after fixing the certificates.

I'll see if we can alert on pods stuck in Pending for a while.

In T358175#9566434, @taavi wrote:

I'll see if we can alert on pods stuck in Pending for a while.

Not easily, the same Pending status as reported by kube-state-metrics seems to also include things pods where the image configured does not exist and other user errors.

Not easily, the same Pending status as reported by kube-state-metrics seems to also include things pods where the image configured does not exist and other user errors.

Does this happen a lot? I would've thought webservice/toolforge jobs would prevent that from happening.

Also, one other thing is that toolforge jobs incorrectly reports the job as "Running for ..." instead of a status of Pending, which you have to use kubectl to see.

In T358175#9570814, @Legoktm wrote:

Not easily, the same Pending status as reported by kube-state-metrics seems to also include things pods where the image configured does not exist and other user errors.

Does this happen a lot? I would've thought webservice/toolforge jobs would prevent that from happening.

There are a few tools currently like this, mostly webservices. The Jobs API does validate it, webservice at the moment does not but that'll be fixed with T348755. And of course people can kubectl directly to create jobs with non-existent images, but hopefully those people know how to check if the image does not exist.

But at least at the moment it's too many to get a meaningful alert that's not mostly triggered by user error.

Also, one other thing is that toolforge jobs incorrectly reports the job as "Running for ..." instead of a status of Pending, which you have to use kubectl to see.

As far as I can tell the job did run successfully, and the error was when it was trying to be removed. But yes, it should report something better in this case. Too bad I didn't save the job YAML before draining the nodes.