[jobs-api] make job status an enum, with clearly defined states
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dcaro
	Aug 5 2025, 8:36 AM

Project Tags

Referenced Files

None

Subscribers

Aklapper

DamianZaremba

dcaro

Description

This task is also to do a first pass of the possible most basic states and use those for starters, this would be different for the three different types of jobs:

ContinuousJob

Pending: from creation until the deployment is healthy
Running: while the deployment is healthy
Failing: if there's any errors/has not started for too long/fails to start
RuntimeError (happy to get other name proposals, InternalError, Inconsistent, Malformed, ...): when the job is missing something on k8s side though it might be running ok (ex. there's the deployment but not the service, or the deployment is missing fields, etc.)
Unknown: any other (details in status_long)

ScheduledJob

This one is a bit special, we can do something like:

LastRunFailed
- Question, do we want to add a LastRunTimedOut?
LastRunPassed
WaitingFirstRun
Failing: this one is in case it's not able to run for whichever reason, maybe collapse with LastRunFailed
Unknown: any other state

OneOffJob

This would be similar to the continuous job:

Pending: until the container starts
Running: from the container start to the end
Success: if it completed ok
Failed: if it failed
Unknown: any other state (details in status_long)

This will allow for using those states in any other parts of the code (ex. cli, web ui, external users like T401069: [jobs-api] report status pending until the first container is running, ...)

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Stalled	BUG REPORT	None	T401069 [jobs-api] report status pending until the first container is running
		Resolved		Raymond_Ndibe	T401172 [jobs-api] make job status an enum, with clearly defined states

Event Timeline

Given the differences is the states, this might be better after T390136: [jobs-api] Split the `*Job` API models into three

dcaro updated the task description. (Show Details)Aug 5 2025, 8:43 AM

dcaro triaged this task as High priority.Aug 5 2025, 11:15 AM

dcaro updated the task description. (Show Details)Aug 5 2025, 11:26 AM

Thanks for outlining this - it would be a super useful addition.

This is perhaps an additional ticket, but exposing the time when the status started would be nice.

For example (assuming T400917 is done):
If the job is "Running" then the logs can be retrieved since the start time - for one-off jobs that re-use a name, this would ensure scoping to a time window of a single run.

If the job is "Failed" then the logs can be retrieved around the failure time, rather than having to fetch all entries.

Currently there is no nice way to know when a job started from the API (I parse the Last run at xxx. Pod in 'Running' phase. message).

Raymond_Ndibe claimed this task.Aug 13 2025, 10:37 AM

Raymond_Ndibe added a project: User-Raymond_Ndibe.

Raymond_Ndibe edited projects, added: Toolforge (Toolforge iteration 23); removed: Toolforge.

Ideally this would be sorted by having system logs, as in logging the system events (ex. job restarted, job stopped, job deleted, ...), the logs-api is starting to take shape, and will unblock the system logs soon-ish.

@Raymond_Ndibe the list of status here is just a proposal, to be discussed/refined, so that's the first part of the task.

In T401172#11082326, @dcaro wrote:

Ideally this would be sorted by having system logs, as in logging the system events (ex. job restarted, job stopped, job deleted, ...), the logs-api is starting to take shape, and will unblock the system logs soon-ish.

unsure what you meant by system logs in this context. Is the plan to do this in logs-api instead? if that's the case I'd rather begin with that instead of doing it on jobs-api and having everything discarded later.

unsure what you meant by system logs in this context. Is the plan to do this in logs-api instead? if that's the case I'd rather begin with that instead of doing it on jobs-api and having everything discarded later.

The "system logs" would be logs that the toolforge system persists for tool users to read, like "restarting job <blabla>" or "deployment <> started", etc.
The idea is to enable an endpoint in logs-api for other toolforge components to register those system event logs, that way the user can go reading the logs and act on those.

In any case, that's a different chunk of work (to be done after the logs-api is around), this is specifically to clarify the possible status each different job can be in, so users can rely on a finite set and act accordingly (that also benefits the toolforge ui, when we want to show different things depending on the state of the job and such...).

I believe status messages should be as uniform as possible. If we need to covey extra information it's better to put that in some form of status detail thing.

status (Replacement for status_short)

I propose the main status enums be just:

pending
running
succeeded
stopped
failed
unknown

status_details (Replacement for status_long)

I propose we start return a structured dictionary for the status detail (what we currently call long). something like this:

{"duration": "5mins", "reason": "anything", "next": "2025-03-04T24:04:05Z"}

duration will likely mean how long the job has been in this state, either in raw seconds in which case the client will need to format it whichever way it chooses. reason is basically any message we need to pass along to explain the status. next only applies to scheduled jobs.

status together with status_details should cover all possible scenarios.

pending:

This will represent all states from when the job is created to when it is running. Any detail about why the job is pending should be added to status_details, that way we have keep track of every context while keeping the status field as simple and streamlined as possible.

one-off | continuous jobs: for one-off jobs this can mean the pod is still waiting to be scheduled, pod scheduled but containers not created, init container still running, etc. All these details will be communicated in status_details. examples:
- pending {"duration": 1000, "reason": "scheduling"} (maybe this can be collapsed into initializing)
- pending {"duration": 1000, "reason": "initializing"}

scheduled jobs: for scheduled jobs this can mean the job is waiting for it's first run, pod scheduled but containers not created, init container still running, etc. All these details will be communicated in status_details. examples:
- pending {"duration": 1000, "reason": "waiting for next run. This can also be omitted since pending + next already conveys this information", "next": "2025-03-04T24:04:05Z"}
- pending {"duration": 1000, "reason": "scheduling", "next": "2025-03-04T24:04:05Z"} (maybe this can be collapsed into initializing)
- pending {"duration": 1000, "reason": "initializing", "next": "2025-03-04T24:04:05Z"}
  
  Notice how we avoided different status messages for scheduled jobs. In a way all status messages describes what happened to a past job (or rather what has happened from a certain point in the past up till now), and in that context, all failed or succeeded for one-off, scheduled and continuous jobs describes what happened to a prev job that has just completed. the next field in status_detail is what differentiates scheduled from others by saying when next will will attempt another run.

running:

self explanatory.

one-off | continuous jobs:: examples:
- running {"duration": 1000, "reason": "running"}

scheduled jobs: examples:
- running {"duration": 1000, "reason": "running", "next": "2025-03-04T24:04:05Z"}

succeeded:

anything from when the job transitioned from running to succeeded to when it goes away or is restarted (for continuous jobs)

one-off | continuous jobs:: examples:
- succeeded {"duration": 1000, "reason": "succeeded | restarting (maybe for continous jobs)"} (do we need to track how many times a job was restarted?)

scheduled jobs: examples:
- succeeded {"duration": 1000, "reason": "succeeded", "next": "2025-03-04T24:04:05Z"}

stopped: (new)

We don't currently have this now, but we'd like to be able to indicate when a job was stopped by the user. This likely means that the job was deleted from the runtime but is still kept in storage.

one-off | continuous jobs:: examples:
- stopped {"duration": 1000, "reason": "stopped"}

scheduled jobs: examples:
- stopped {"duration": 1000, "reason": "stopped", "next": "2025-03-04T24:04:05Z"}

failed:

this means something is wrong with the job.

one-off | continuous jobs:: examples:
- failed {"duration": 1000, "reason": "some reason, quota, etc"}

scheduled jobs: examples:
- failed {"duration": 1000, "reason": "some reason, quota, etc", "next": "2025-03-04T24:04:05Z"}

unknown:

If for some reason we can't get the status of the job, this will be returned. If we have an idea why we can't get the status, we can explain that in status_detail.

one-off | continuous jobs:: examples:
- unknown {"duration": 1000, "reason": "unknown reason"}

scheduled jobs: examples:
- unknown {"duration": 1000, "reason": "unknown reason", "next": "2025-03-04T24:04:05Z"}

The Inconsistent State

I avoided this above because it is a special case in the sense that it is something that can apply to all jobs, in all states, so isn't really a specific state on it's own. A job can be in pending | running | failing | unknown and the saved spec will still be out of sync with the running spec in the runtime (inconsistent or whatever we want to call this). It's not an error either in the same way we view errors, because it doesn't mean a job is not running and doing stuff.
For this reason I think this belongs either in status_detail as it's own unique field or we can convert reason to messages, make it an array and put stuffs inside .e.g. pending {"duration": 1000, "messages": ["initializing", "inconsistent"], "next": "2025-03-04T24:04:05Z"}.
Suggestions are also welcome.

fnegri mentioned this in T401069: [jobs-api] report status pending until the first container is running.Aug 20 2025, 2:19 PM

After a live discussion, we agreed to the above with the following minor changes:

Have a single status with the whole sturcutre, and a short entry with the short status:

"status": {
    "short": "running",
    ...
}

Move the inconsistent as it's own property and rename to up_to_date (this was not discussed, but I propose it @Raymond_Ndibe):

"status": {
   ...
   "up_to_date": true,
   ...
}

So for continuous jobs would be:

# DefinedContinuousJob
{
    ...
    "status": {
        "short": "running",
        "messages": ["container running"],
        "duration": "00:00:32",
        "up_to_date": false,
    }
    ...
}

For scheduled job:

# DefinedScheduledJob
{
    ...
    "status": {
        "short": "failed",
        "messages": ["Out of memory"],
        "duration": "00:00:32",
        "last_run_time": "2025-03-14 00:01:32",
        "next_run_time": "2025-04-32 00:00:01",
        "last_success_time": "2025-02-23 00:04:12",
        "up_to_date": true
    }
    ...
}

For one-off jobs:

# DefinedOneoffJob
{
    ...
    "status": {
        "short": "failed",
        "messages": ["Command not found"],
        "duration": "00:00:32",
        "last_run_time": "2025-03-14 00:01:32",
        "up_to_date": false
    }
    ...
}

The meanining of running (for all the job types) is: When the container is running and *healthy (if it has healthchecks)

The meaning of pending, for one-off/continuous is: Anything between job registered and not yet running

For scheduled jobs the meaning of pending` is: The schedule is registered (cronjob created in k8s), and waiting for the first run to be running

The succeeded status only applies to scheduled/one-off, and it means that the last run

Implementation plan

add the new status field and flag status_short and status_long as deprecated pointing to the new structure
- the status field with all the statuses except stopped, without duration, and without up_to_date
Adding the duration field
Adding the stopped short state (when we have storage)
Adding the up_to_date field (when we have storage)

@Raymond_Ndibe Does that sound correct? (if so I'll create the tasks)

This sounds like a good improvement.

Just a question regarding inconsistent/up_to_date - I can't quite parse "the saved spec will still be out of sync with the running spec in the runtime", is the intention for this to reflect:

Job config sent to API has not been synced to Job object in runtime (k8s) - I think this is done sync during the API call?
Job instance running (pod) is using an older version of spec than is in Job (k8s object) i.e. it was started before the Job changed - Continuous would be restarted, One off would just exit so this would only really apply to Scheduled?

In T401172#11110418, @DamianZaremba wrote:

This sounds like a good improvement.

Just a question regarding inconsistent/up_to_date - I can't quite parse "the saved spec will still be out of sync with the running spec in the runtime", is the intention for this to reflect:

Job config sent to API has not been synced to Job object in runtime (k8s) - I think this is done sync during the API call?

Job instance running (pod) is using an older version of spec than is in Job (k8s object) i.e. it was started before the Job changed - Continuous would be restarted, One off would just exit so this would only really apply to Scheduled?

@DamianZaremba so when a user runs toolforge jobs run, the job is sent to the jobs-api. on the jobs-api the job is passed through several functions and methods that in turn may make some modifications to the initial job submitted by the user, before finally being sent off to k8s to create the underlying k8s resource. To get details of a running job this flow is reversed.

Now the problem with this is that during this process we lose the exact job submitted by the user and it's a bit error prone and difficult to get this back from k8s resources alone, but this is what we currently do with varying degree of success. To solve this problem we plan to persist the exact job submitted by the user in some db, before sending it to k8s to run, thereby clearly separating storage from runtime.
The problem with this second approach is that, because the cronjobs and jobs and deployments run in the tools namespace and is accessible to the tool owner when they become the tool, the resources running in the runtime can potentially be manually edited. We intend to use the up_to_date field to communicate this potential drift between what is in storage and what is running in the runtime.

one-off | continuous jobs: examples:
- {"short": "pending", "messages": ["restarting, maybe retrying?"], "duration": "00:00:32", "up_to_date": false} for jobs that are restarting either because of failure when backofflimit is specified (for jobs), or the restarting if command has exited (for deployments).
- {"short": "pending", "messages": ["scheduling"], "duration": "00:00:32", "up_to_date": false} pod is waiting to be assigned to node
- {"short": "pending", "messages": ["initializing"], "duration": "00:00:32", "up_to_date": false} pod init containers are still running, images still getting pulled
- {"short": "running", "messages": ["running"], "duration": "00:00:32", "up_to_date": true} all containers in the pod are running
- {"short": "succeeded", "messages": ["succeeded"], "duration": "00:00:32", "up_to_date": true} pod containers exited successfully
- {"short": "stopped", "messages": ["stopped"], "duration": "00:00:32", "up_to_date": true} (upcoming) job was stopped by user, to maybe be restarted later.
- {"short": "failed", "messages": ["Command not found"], "duration": "00:00:32", "up_to_date": true} the pod, container(s) failed to run
- {"short": "unknown", "messages": ["unknown"], "duration": "00:00:32", "up_to_date": true} unable to get the status of the job for some reason

scheduled jobs: examples:
- {"short": "pending", "messages": ["restarting, maybe retrying?"], "duration": "00:00:32", "last_run_time": "2025-03-14 00:01:32", "next_run_time": "2025-04-32 00:00:01", "last_success_time": "2025-02-23 00:04:12", "up_to_date": true} for jobs that are restarting either of failure when backofflimit is specified
- {"short": "pending", "messages": ["scheduling"], "duration": "00:00:32", "last_run_time": "2025-03-14 00:01:32", "next_run_time": "2025-04-32 00:00:01", "last_success_time": "2025-02-23 00:04:12", "up_to_date": true} pod is waiting to be assigned to node
- {"short": "pending", "messages": ["initializing"], "duration": "00:00:32", "last_run_time": "2025-03-14 00:01:32", "next_run_time": "2025-04-32 00:00:01", "last_success_time": "2025-02-23 00:04:12", "up_to_date": true} ("last_run_time" and "last_success_time" being null means the job is still waiting for it's first run) pod init containers are still running, images still getting pulled
- {"short": "running", "messages": ["running"], "duration": "00:00:32", "last_run_time": "2025-03-14 00:01:32", "next_run_time": "2025-04-32 00:00:01", "last_success_time": "2025-02-23 00:04:12", "up_to_date": true} all containers in the pod are running
- {"short": "succeeded", "messages": ["succeeded"], "duration": "00:00:32", "last_run_time": "2025-03-14 00:01:32", "next_run_time": "2025-04-32 00:00:01", "last_success_time": "2025-02-23 00:04:12", "up_to_date": true} pod containers exited successfully
- {"short": "stopped", "messages": ["stopped"], "duration": "00:00:32", "last_run_time": "2025-03-14 00:01:32", "next_run_time": "2025-04-32 00:00:01", "last_success_time": "2025-02-23 00:04:12", "up_to_date": true} job was stopped by user, to maybe be restarted later.
- {"short": "failed", "messages": ["Command not found"], "duration": "00:00:32", "last_run_time": "2025-03-14 00:01:32", "next_run_time": "2025-04-32 00:00:01", "last_success_time": "2025-02-23 00:04:12", "up_to_date": true} the pod, container(s) failed to run
- {"short": "unknown", "messages": ["unknown"], "duration": "00:00:32","last_run_time": "2025-03-14 00:01:32", "next_run_time": "2025-04-32 00:00:01", "last_success_time": "2025-02-23 00:04:12", "up_to_date": true} unable to get the status of the job for some reason