Move Kubernetes log source multi-pod handling from jobs-api to toolforge-weld
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	taavi
	Jul 3 2025, 4:11 PM

Description

T388274: [jobs-api] "toolforge jobs logs <name> -f" should get the logs of all containers in all target pods added logic in jobs-api to handle loading logs from the Kubernetes logging API in instances where there are multiple pods. I think this logic should live in toolforge-weld instead, as:

that way webservice logs can also take advantage of this bugfix
this lets jobs-api to use a non-implementation-specific interface (LogSource.query(), that is) so that in the parent task it can be swapped to Loki much more easily.

Details

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
jobs-api: bump to 0.0.386-20250729091039-7e54536b	repos/cloud/toolforge/toolforge-deploy!884	group_203_bot_f4d95069bb2675e4ce1fff090c1c1620	bump_jobs-api	main
Use logging multi-pod fix moved to toolforge-weld	repos/cloud/toolforge/jobs-api!179	taavi	taavi/logging	main
logs: Move multi-pod fix from jobs-api to here	repos/cloud/toolforge/toolforge-weld!82	taavi	taavi/logs-multi-pod	main

Customize query in GitLab

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		• Bstorm	T126083 overhaul labstore setup [tracking]
Resolved		• GTirloni	T216988 labstore1004 - DISK CRITICAL - free space: /srv/tools 115904 MB (1% inode=79%):
Resolved		• Bstorm	T217993 2019-03-10: tools and NFS share cleanup (high usage)
Resolved		• Bstorm	T122508 Prevent overly-large log files
Declined		None	T286847 Add webservice flag to mount project directory read-only
Open		taavi	T127367 [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools
Resolved	Feature	taavi	T400913 [jobs-api] Allow querying logs for non-existent jobs
Open		None	T400914 [jobs-api] Remove file logging support
Open		None	T400915 [jobs-api] Disable file logging by default
Resolved		taavi	T400916 [jobs-api] Support following logs from Loki
Open	Feature	Raymond_Ndibe	T400917 [jobs-api] Allow customizing time to request Loki logs for
Resolved		taavi	T398645 [jobs-api] Jobs API should query logs from Loki
Resolved		taavi	T398647 Move Kubernetes log source multi-pod handling from jobs-api to toolforge-weld

Event Timeline

taavi created this task.Jul 3 2025, 4:11 PM

taavi triaged this task as Medium priority.

I have my doubts about this, as relatively soon we will move webservice to jobs-api, and then there will be no other users of the logging code making it better to not have in an external library, imo it's ok to move it to the jobs-api directly instead.

Yeah, most of this will soon be obsolete. But right now it isn't, and I don't want to block the Loki work on the webservice->jobs migration, and this is very much a similar amount of work compared to the alternative (crafting a "wrapper" LogSource in jobs-api internally), and it gives webservice users a better experience in the mean time.

In T398647#10973253, @taavi wrote:

Yeah, most of this will soon be obsolete. But right now it isn't, and I don't want to block the Loki work on the webservice->jobs migration, and this is very much a similar amount of work compared to the alternative (crafting a "wrapper" LogSource in jobs-api internally), and it gives webservice users a better experience in the mean time.

👍

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/82

logs: Move multi-pod fix from jobs-api to here

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/179

Draft: Use logging multi-pod fix moved to toolforge-weld

taavi edited projects, added Toolforge (Toolforge iteration 21); removed Toolforge.Jul 4 2025, 10:02 AM

taavi moved this task from Next Up to In Review on the Toolforge (Toolforge iteration 21) board.

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/82

logs: Move multi-pod fix from jobs-api to here

dcaro edited projects, added Toolforge (Toolforge iteration 22); removed Toolforge (Toolforge iteration 21).Jul 14 2025, 9:47 AM

dcaro moved this task from Next Up to In Review on the Toolforge (Toolforge iteration 22) board.

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/179