[jobs-api] Support following logs from Loki
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	taavi
	Jul 31 2025, 3:54 PM

Description

Port the jobs logs --follow feature to read logs from Loki instead of from Kubernetes.

This is hard because the Loki API for this uses websockets, and the modern Python websocket client libraries prefer to use asyncio.

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	jobs-api: bump to 0.0.395-20250806171229-63780c9e	repos/cloud/toolforge/toolforge-deploy!910	group_203_bot_f4d95069bb2675e4ce1fff090c1c1620	bump_jobs-api	main
	loki_logs: Support following logs	repos/cloud/toolforge/jobs-api!193	taavi	taavi/loki-follow	main

Status	Assigned	Task
Resolved	• Bstorm	T126083 overhaul labstore setup [tracking]
Resolved	• GTirloni	T216988 labstore1004 - DISK CRITICAL - free space: /srv/tools 115904 MB (1% inode=79%):
Resolved	• Bstorm	T217993 2019-03-10: tools and NFS share cleanup (high usage)
Resolved	• Bstorm	T122508 Prevent overly-large log files
Declined	None	T286847 Add webservice flag to mount project directory read-only
Open	taavi	T127367 [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools
Resolved	taavi	T400916 [jobs-api] Support following logs from Loki
Resolved	taavi	T398645 [jobs-api] Jobs API should query logs from Loki
Resolved	taavi	T386480 [o11y,logging,infra] Deploy Loki to store Toolforge tool log data
Resolved	taavi	T396574 Provision object storage volumes for Loki
Resolved	Andrew	T396594 Create OpenStack role that allows object storage access only
Resolved	dcaro	T397339 Request creation of toolsbeta-logging VPS project
Resolved	None	T397446 Request creation of tools-logging VPS project
Resolved	Andrew	T398447 Increase tools-logging object storage quotas
Resolved	taavi	T398647 Move Kubernetes log source multi-pod handling from jobs-api to toolforge-weld
Resolved	taavi	T400795 [jobs-api] 400 bad request when trying to load a large number of logs from Loki
Resolved	taavi	T401113 [jobs-api] Migrate to FastAPI