Steps to replicate the issue (include links if applicable):
$ toolforge jobs run --image bookworm --command "exit 0" run-test; toolforge jobs show run-test | grep Status; kubectl get pods | grep run-test- | Status: | Running for 1s | run-test-m8l9h 0/1 ContainerCreating 0 0s
What happens?:
_get_job_object_status (https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/blob/main/tjf/runtimes/k8s/ops_status.py#L36) returns "Running" as soon as the Job is active & has a startTime.
As the job becomes active before the container starts running, the API/CLI shows the user that the job is running, when actually it isn't, it is pending/initializing. There is regularly a few seconds between the API reporting "Running" and any actual command execution.
What should have happened instead?:
The status should be reported as pending/initializing/waiting until both the Pod and Container are created.
In the case that a health-check is defined on the Container then either the status should reflect the condition of this or an additional health field should be returned.
Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):
$ toolforge --version Toolforge CLI, version 0.3.6
Other information (browser name/version, screenshots, etc.):
Currently after launching a job using the Jobs API, I query the Kubernetes API directly to wait until the container is launched and the Readiness is health.
This is a little clunky as certain details, such as the kubernetes namespace, are not directly exposed (the tools API works on tool name, kubernetes on a prefixed tool name).