Page MenuHomePhabricator

[jobs-api] 400 bad request when trying to load a large number of logs from Loki
Closed, ResolvedPublic

Description

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/toolforge_weld/api_client.py", line 136, in _make_request
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: https://api.svc.toolsbeta.eqiad1.wikimedia.cloud:30003/jobs/v1/tool/taavi-test/jobs/test/logs?follow=false&lines=10000

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/tjf_cli/cli.py", line 1091, in main
    run_subcommand(args=args, api=api, toolname=toolname)
  File "/usr/lib/python3/dist-packages/tjf_cli/cli.py", line 1013, in run_subcommand
    op_logs(api, args.name, args.follow, args.last)
  File "/usr/lib/python3/dist-packages/tjf_cli/cli.py", line 723, in op_logs
    for raw_line in api.get_raw_lines(
  File "/usr/lib/python3/dist-packages/toolforge_weld/api_client.py", line 214, in get_raw_lines
    with self._make_request(
         ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/toolforge_weld/api_client.py", line 156, in _make_request
    raise exception_object from e
tjf_cli.api.TjfCliError: Unknown error (400 Client Error: Bad Request for url: http://loki-tools-read.loki.svc.toolsbeta.local:3100/loki/api/v1/query_range?query=%7Bjob%3D%22test%22%7D&since=1h&limit=10000&direction=forward)
[2025-07-30 09:43:27] [errors.py] ERROR: Some additional context for the issue follows:
[2025-07-30 09:43:27] [errors.py] ERROR:  messages = {"error": ["Unknown error (400 Client Error: Bad Request for url: http://loki-tools-read.loki.svc.toolsbeta.local:3100/loki/api/v1/query_range?query=%7Bjob%3D%22test%22%7D&since=1h&limit=10000&direction=forward)"]}

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
jobs-api: bump to 0.0.389-20250731145816-0cabba30repos/cloud/toolforge/toolforge-deploy!890group_203_bot_f4d95069bb2675e4ce1fff090c1c1620bump_jobs-apimain
loki_logs: Raise user-friendly error when exceeding line limitrepos/cloud/toolforge/jobs-api!188taavitaavi/lokimain
Customize query in GitLab

Event Timeline

taavi triaged this task as High priority.

The error from Loki is:

max entries limit per query exceeded, limit > max_entries_limit_per_query (10000 > 5000)
taavi changed the task status from Open to In Progress.Jul 30 2025, 9:57 AM
taavi moved this task from Next Up to In Review on the Toolforge (Toolforge iteration 22) board.

group_203_bot_f4d95069bb2675e4ce1fff090c1c1620 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/890

jobs-api: bump to 0.0.389-20250731145816-0cabba30