Note that I’m still using toolforge: tool in other Kubernetes objects, where it seems to work; in particular, the restart cronjob runs twice a day, successfully (kubectl get deployment function-evaluator -o json | jq -r '.spec.template.metadata.annotations["kubectl.kubernetes.io/restartedAt"]' shows that the function-evaluator deployment was actually restarted), and to do so it must have the home directory mounted (otherwise kubectl would be missing its credentials).

(The other toolforge: tool object I have is the update deployment, but that one has been running since before PodPresets were phased out, if I’m not mistaken, so that doesn’t necessarily prove anything.)

On both pods for the controller I see on of these on different dates: 2021/09/30 17:22:52 http: TLS handshake error from 192.168.48.128:39842: EOF, but it seems to be functioning for the most part, so I'm not sure what that's about. That's not currently the pod's IP address, and I don't even see that IP in the current environment, so I presume that's just some old stuff.

I was able to mount things when I ran a quick "webservice shell" to simulate using the preset. The "missing path" error sounds like it is trying to patch the wrong object almost, but the definition doesn't look wrong. I don't see the error just yet. I'll poke a little more.

I see your problem with the new setup:

automountServiceAccountToken: false

By not mounting the SA token, there's no "volumes" item in your pod. If there's no volumes item, you cannot do an "add" operation, which is what the mutating webhook controller does.

For now, I am willing to bet, you can just remove that line, and your tool will work via the label again. We also should probably upgrade the web hook so that it can function without a pre-existing volumes list as well.

To be really specific, this line: https://github.com/lucaswerkmeister/notwikilambda-k8s/blob/cef7948b15922240cf113dc1e7621ee74715d95d/function-orchestrator/deployment.yaml#L21

Very similar to what people experienced here https://github.com/evanphx/json-patch/issues/138. The json representation of the structure of the pod object is not fixed in k8s, so if there is no volume at all on the pod, this mutator fails because it is using json-patch as it's strategy. Since the default service account mounts its own creds in most pods, this isn't a problem unless you create your own object that disables that feature.

In T292672#7406964, @Bstorm wrote:

I see your problem with the new setup:

automountServiceAccountToken: false

By not mounting the SA token, there's no "volumes" item in your pod. If there's no volumes item, you cannot do an "add" operation, which is what the mutating webhook controller does.

https://github.com/wikimedia/cloud-toolforge-volume-admission-controller/blob/0a2d76cf09541767df7d675cf7d7e8a88c885e6b/server/admission.go#L148
I was curious how the admission controller was adding volumes to a non-existent array. So the SA automount is the answer.

I've got a patch set that I'm going to test in Minikube in a few.

Change 726997 had a related patch set uploaded (by Bstorm; author: Bstorm):

[cloud/toolforge/volume-admission-controller@main] json-patch: create empty arrays where needed

https://gerrit.wikimedia.org/r/726997

Change 726997 merged by jenkins-bot:

[cloud/toolforge/volume-admission-controller@main] json-patch: create empty arrays where needed

https://gerrit.wikimedia.org/r/726997

Mentioned in SAL (#wikimedia-cloud) [2021-10-07T05:44:42Z] <majavah> deploying fix for T292672