It should be possible for a job/service inside k8s to start/restart another job or service. Currently this is not possible without invoking the internal jobs API as the container images don't have toolforge-jobs cli.

Usecases:

• A lightweight web-facing application may want to defer some heavy processing off to a separate pod to reduce resource consumption on the main service.
• Sometimes, a cronjob is setup to check whether another service is running (e.g. by reading the log file which the service is supposed to update). If the service is found to be not running (despite the pod being active, such as due to a stuck process), the cronjob should be able to restart the service.

Possible solutions:

• Rewrite toolforge-jobs cli in a language that compiles to machine code (Go/Rust/C++) and install the binary in all images.
• Make the toolforge-jobs-api service stable and non-internal, allowing tools to access it directly?
• ...