Page MenuHomePhabricator

[jobs-api] report status pending until the first container is running
Open, Stalled, MediumPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

$ toolforge jobs run --image bookworm --command "exit 0" run-test; toolforge jobs show run-test | grep Status; kubectl get pods | grep run-test-
| Status:       | Running for 1s                                                  |
run-test-m8l9h                       0/1     ContainerCreating   0          0s

What happens?:

_get_job_object_status (https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/blob/main/tjf/runtimes/k8s/ops_status.py#L36) returns "Running" as soon as the Job is active & has a startTime.

As the job becomes active before the container starts running, the API/CLI shows the user that the job is running, when actually it isn't, it is pending/initializing. There is regularly a few seconds between the API reporting "Running" and any actual command execution.

What should have happened instead?:

The status should be reported as pending/initializing/waiting until both the Pod and Container are created.

In the case that a health-check is defined on the Container then either the status should reflect the condition of this or an additional health field should be returned.

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):

$ toolforge --version
Toolforge CLI, version 0.3.6

Other information (browser name/version, screenshots, etc.):

Currently after launching a job using the Jobs API, I query the Kubernetes API directly to wait until the container is launched and the Readiness is health.
This is a little clunky as certain details, such as the kubernetes namespace, are not directly exposed (the tools API works on tool name, kubernetes on a prefixed tool name).

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Playing this this a bit more, you can sort of get the container status by parsing the long status string.

Maybe the solution here is to expose status / status_long in a more structured way.

https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/blob/main/tjf_cli/cli.py?ref_type=heads#L564 handles this by not checking for running, but checking completion.. I can do something similar if needed

dcaro renamed this task from Toolforge jobs api reports status as running before container is launched to [jobs-api] report status pending until the first container is running.Aug 5 2025, 8:25 AM
fnegri changed the task status from Open to Stalled.Aug 20 2025, 2:19 PM
fnegri triaged this task as Medium priority.
fnegri subscribed.

T401172: [jobs-api] make job status an enum, with clearly defined states relates to this and is currently being worked on, we can revisit this task after that one is resolved.