Steps to replicate the issue (include links if applicable):
- Run a one-off job that produces output (log entries)
- Query the logs for the job as it is running / shortly after it has finished
- Observe no log output (API returns 404)
What happens?:
The toolforge logs command returns no logs for a job while it is running, or shortly afterwards.
After a period of time, the logs are returned via the API.
A minimal reproduction:
tools.cluebotng-trainer@tools-bastion-13:~$ toolforge jobs run --image bookworm --command 'date; exit 1' --wait 5 --no-filelog test-logging; toolforge jobs logs test-logging ERROR: job 'test-logging' failed: +---------------+------------------------------------------------------------------+ | Job name: | test-logging | +---------------+------------------------------------------------------------------+ | Command: | date; exit 1 | +---------------+------------------------------------------------------------------+ | Job type: | one-off | +---------------+------------------------------------------------------------------+ | Image: | bookworm | +---------------+------------------------------------------------------------------+ | Port: | none | +---------------+------------------------------------------------------------------+ | File log: | no | +---------------+------------------------------------------------------------------+ | Output log: | | +---------------+------------------------------------------------------------------+ | Error log: | | +---------------+------------------------------------------------------------------+ | Emails: | none | +---------------+------------------------------------------------------------------+ | Resources: | mem: 0.5Gi, cpu: 0.5 | +---------------+------------------------------------------------------------------+ | Replicas: | | +---------------+------------------------------------------------------------------+ | Mounts: | all | +---------------+------------------------------------------------------------------+ | Retry: | no | +---------------+------------------------------------------------------------------+ | Timeout: | no | +---------------+------------------------------------------------------------------+ | Health check: | none | +---------------+------------------------------------------------------------------+ | Status: | Failed | +---------------+------------------------------------------------------------------+ | Hints: | Last run at 2025-08-24T13:33:36Z. Pod in 'Failed' phase. State | | | 'terminated'. Reason 'Error'. Started at '2025-08-24T13:33:37Z'. | | | Finished at '2025-08-24T13:33:37Z'. Exit code '1'. | +---------------+------------------------------------------------------------------+ ERROR: Job 'test-logging' does not have any logs available tools.cluebotng-trainer@tools-bastion-13:~$ tools.cluebotng-trainer@tools-bastion-13:~$ date; toolforge jobs logs test-logging Sun Aug 24 13:33:53 UTC 2025 2025-08-24T13:33:37+00:00 [test-logging-d9x7s] [job] Sun Aug 24 01:33:37 PM UTC 2025
What should have happened instead?:
Logs should be streamed from the container, this is the previous behaviour.