Page MenuHomePhabricator

toolforge logs appears to suffer from intermittent latency
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  • Run a one-off job that produces output (log entries)
  • Query the logs for the job as it is running / shortly after it has finished
  • Observe no log output (API returns 404)

What happens?:

The toolforge logs command returns no logs for a job while it is running, or shortly afterwards.

After a period of time, the logs are returned via the API.

A minimal reproduction:

tools.cluebotng-trainer@tools-bastion-13:~$ toolforge jobs run --image bookworm --command 'date; exit 1' --wait 5 --no-filelog test-logging; toolforge jobs logs test-logging
ERROR: job 'test-logging' failed:
+---------------+------------------------------------------------------------------+
| Job name:     | test-logging                                                     |
+---------------+------------------------------------------------------------------+
| Command:      | date; exit 1                                                     |
+---------------+------------------------------------------------------------------+
| Job type:     | one-off                                                          |
+---------------+------------------------------------------------------------------+
| Image:        | bookworm                                                         |
+---------------+------------------------------------------------------------------+
| Port:         | none                                                             |
+---------------+------------------------------------------------------------------+
| File log:     | no                                                               |
+---------------+------------------------------------------------------------------+
| Output log:   |                                                                  |
+---------------+------------------------------------------------------------------+
| Error log:    |                                                                  |
+---------------+------------------------------------------------------------------+
| Emails:       | none                                                             |
+---------------+------------------------------------------------------------------+
| Resources:    | mem: 0.5Gi, cpu: 0.5                                             |
+---------------+------------------------------------------------------------------+
| Replicas:     |                                                                  |
+---------------+------------------------------------------------------------------+
| Mounts:       | all                                                              |
+---------------+------------------------------------------------------------------+
| Retry:        | no                                                               |
+---------------+------------------------------------------------------------------+
| Timeout:      | no                                                               |
+---------------+------------------------------------------------------------------+
| Health check: | none                                                             |
+---------------+------------------------------------------------------------------+
| Status:       | Failed                                                           |
+---------------+------------------------------------------------------------------+
| Hints:        | Last run at 2025-08-24T13:33:36Z. Pod in 'Failed' phase. State   |
|               | 'terminated'. Reason 'Error'. Started at '2025-08-24T13:33:37Z'. |
|               | Finished at '2025-08-24T13:33:37Z'. Exit code '1'.               |
+---------------+------------------------------------------------------------------+
ERROR: Job 'test-logging' does not have any logs available

tools.cluebotng-trainer@tools-bastion-13:~$ 
tools.cluebotng-trainer@tools-bastion-13:~$ date; toolforge jobs logs test-logging
Sun Aug 24 13:33:53 UTC 2025
2025-08-24T13:33:37+00:00 [test-logging-d9x7s] [job] Sun Aug 24 01:33:37 PM UTC 2025

What should have happened instead?:

Logs should be streamed from the container, this is the previous behaviour.

Event Timeline

As a workaround I have the job emitting a known log line and then poll the logs API until that line in seen, this isn't ideal as it adds extra resource usage to both the job coordinator and the logs API.

fgiunchedi triaged this task as Medium priority.Sep 1 2025, 2:40 PM

Example occurrences:

Timed out after 2min waiting for the log entries:

2025-08-24 16:54:38,388 [INFO] [coord-random-edits] Job succeeded
2025-08-24 16:56:39,457 [ERROR] [coord-random-edits] Timed out before log end marker

Got the log entries immediately:

2025-08-24 16:57:48,705 [INFO] [coord-false-negatives-2] Job succeeded
2025-08-24 16:57:49,109 [INFO] [coord-false-negatives-2] Found log end marker

Today a job managed to get partial logs... not the start, not the end, but something in the middle =\

tools.cluebotng@tools-bastion-15:~$ toolforge jobs logs -l 5000 report-review-import
2025-09-26T18:13:14+00:00 [report-review-import-29315173-wrdh2] [job] [
2025-09-26T18:54:51+00:00 [report-review-import-zcd8m] [job]             "2324296": 5,
2025-09-26T18:54:52+00:00 [report-review-import-zcd8m] [job]             "2735569": 5,
2025-09-26T18:54:52+00:00 [report-review-import-zcd8m] [job]             "3220885": 6,
2025-09-26T18:54:52+00:00 [report-review-import-zcd8m] [job]             "856099500": 5,
2025-09-26T18:54:52+00:00 [report-review-import-zcd8m] [job]             "973254056": 6,
2025-09-26T18:54:52+00:00 [report-review-import-zcd8m] [job]             "639289933": 6,
2025-09-26T18:54:52+00:00 [report-review-import-zcd8m] [job]             "681765410": 6,
2025-09-26T18:54:52+00:00 [report-review-import-zcd8m] [job]             "1303094942": 5,
2025-09-26T18:54:52+00:00 [report-review-import-zcd8m] [job]             692551823,
2025-09-26T18:54:52+00:00 [report-review-import-zcd8m] [job]             1030249651,

That is suppose to be a json structure (see https://cluebotng.toolforge.org/api/?action=review.import) and is defiantly always more than 11 lines.

The execution was successful, just the output was mangled.

I think that might be the rate limiting happening, as alloy did pick up the logfile, but it seems the logs did not reach the storage, maybe we have to look into how to tell alloy to retry them (if that's the case)

As of right now, running this returns the first 100 logs, then one extra for some reason:

tools.wm-lol@tools-bastion-15:~$ cat test.sh 
#!/bin/bash

for i in $(seq 6000); do
    echo "line $i"
done


tools.wm-lol@tools-bastion-15:~$ toolforge jobs run --command '$TOOL_DATA_DIR/test.sh' --image python3.11 --no-filelog testjob1

tools.wm-lol@tools-bastion-15:~$ toolforge jobs logs -f testjob1
2025-09-29T14:11:26+00:00 [testjob1-mxt4k] [job] line 1
2025-09-29T14:11:26+00:00 [testjob1-mxt4k] [job] line 2
2025-09-29T14:11:26+00:00 [testjob1-mxt4k] [job] line 3
2025-09-29T14:11:26+00:00 [testjob1-mxt4k] [job] line 4
2025-09-29T14:11:26+00:00 [testjob1-mxt4k] [job] line 5
2025-09-29T14:11:26+00:00 [testjob1-mxt4k] [job] line 6
2025-09-29T14:11:26+00:00 [testjob1-mxt4k] [job] line 7
... all good here
2025-09-29T14:11:26+00:00 [testjob1-mxt4k] [job] line 99
2025-09-29T14:11:26+00:00 [testjob1-mxt4k] [job] line 100
2025-09-29T14:11:26+00:00 [testjob1-mxt4k] [job] line 3322

A different run:

...
2025-09-29T14:13:31+00:00 [testjob1-rjw2m] [job] line 100
2025-09-29T14:13:31+00:00 [testjob1-rjw2m] [job] line 2787
... this one got only to 45
2025-09-29T14:14:00+00:00 [testjob1-26ssm] [job] line 45
2025-09-29T14:14:00+00:00 [testjob1-26ssm] [job] line 562
2025-09-29T14:14:00+00:00 [testjob1-26ssm] [job] line 4879

I can reproduce on lima-kilo too, pointing too to the limit being reached instead of being something deeployment-related (cluster load/network/etc.).

From alloy logs for one of the runs:

│ ts=2025-09-29T14:35:17.935750637Z level=info msg="tail routine: started" component_path=/ component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/tool-tf-test_testjob1-vbw5k_c5d39240-9755-434b-8178-d1563ce440c8/job/0.log                                                                                                     │
│ ts=2025-09-29T14:35:17.935839251Z level=info msg="Seeked /var/log/pods/tool-tf-test_testjob1-vbw5k_c5d39240-9755-434b-8178-d1563ce440c8/job/0.log - &{Offset:0 Whence:0}" component_path=/ component_id=loki.source.file.pod_logs component=tailer                                                                                                  │
│ ts=2025-09-29T14:35:47.934248756Z level=info msg="tail routine: tail channel closed, stopping tailer" component_path=/ component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/tool-tf-test_testjob1-vbw5k_c5d39240-9755-434b-8178-d1563ce440c8/job/0.log reason=<nil>                                                           │
│ ts=2025-09-29T14:35:47.93432277Z level=info msg="tail routine: exited" component_path=/ component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/tool-tf-test_testjob1-vbw5k_c5d39240-9755-434b-8178-d1563ce440c8/job/0.log                                                                                                       │
│ ts=2025-09-29T14:35:47.934342547Z level=info msg="position timer: exited" component_path=/ component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/tool-tf-test_testjob1-vbw5k_c5d39240-9755-434b-8178-d1563ce440c8/job/0.log                                                                                                    │
│ ts=2025-09-29T14:35:47.934376517Z level=info msg="stopped tailing file" component_path=/ component_id=loki.source.file.pod_logs component=tailer path=/var/log/pods/tool-tf-test_testjob1-vbw5k_c5d39240-9755-434b-8178-d1563ce440c8/job/0.log

I see two issues:

  • the rate limiting prevents us from sending more than 100 logs at a time, and the current policy in alloy config is to drop if that's reached, so it just does not retry
  • when not using --follow (default) it seems to get N logs from the start of the time period of 1h, and then sort them from newest to oldest, instead of returning the last N logs of the time slot (what I would have expected)

Looking into both as separate issues.

the rate limiting prevents us from sending more than 100 logs at a time

While perhaps not great for real time troubleshooting, you can (at least today) get the pods logs when you need a short cycle between adjusting something and seeing the logs.

Given it's a shared service and does not currently auto scale, having a rate limit is generally a good idea.

the current policy in alloy config is to drop if that's reached, so it just does not retry

This is pretty terrible from the maintainers side, especially combined with the above.

In the best case it makes logging unreliable and in the worst case, during the moments when you have increased logs (i.e. an issue), they get dropped.

Another slant to rate limiting/rety (e.g. delay) is how often are the workers being killed vs gracefully drained (assuming alloy is wired to flush logs when a node is coord)?

For nfs the nodes lock up quite often and get rebooted - i assume this is hard, so likely logs are also not flushed.

Today it's a choice between not having logs (1hr look back, dropping) and been able to use the non nfs workers (which are more stable) or using file logging and getting containers stuck. I'm sure this is know, but I'm just outlining the less than ideal situation as a tool maintainer

dcaro changed the task status from Open to In Progress.Oct 13 2025, 5:33 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 24) board.

The rate limiting has been tweaked now, it should not be dropping as many logs, but instead just go sending them at the rate limit until done or the logs become older than the retention.

Will have to look into the capacity and retention the next days to make sure we don't run out of space.

dcaro moved this task from In Progress to Done on the Toolforge (Toolforge iteration 25) board.

I think we can declare this fixed, we'll open a task if anyone finds issues again.
The object store capacity is more or less stable at around 65%