I attempted to create a cron job that restarts a k8s pod (nevermind why!). I get the error kubectl: not found. It apparently doesn't know about kubernetes.
Cron is something like:
*/10 * * * * jlocal ~/restart_pod.sh
And restart_pod.sh:
#!/bin/bash
kubectl delete deployment <deployment-name>
kubectl create -f /data/project/<tool>/deployment.yamland is executable.
I think we talked about this on irc at the time this bug was filed, but the toolforge cron server is not currently setup with the role::toollabs::k8s::bastion Puppet role that would make this possible.
I'm not sure when this was fixed, but both tools-cron-01.tools.eqiad.wmflabs and tools-sgecron-01.tools.eqiad.wmflabs have webservice installed.
We try not to reuse hostnames so if we build a new instance to replace an old one we will generally increment the instance number at the end of the hostname. Usually something-01 and something-02 are interchangeable from the point of view of the end user. Sometimes the instances will be running as a hot-cold or hot-warm cluster where a key service is only active on one instance or the other at any given time.