[jobs-api] "toolforge jobs logs <name> -f" should get the logs of all containers in all target pods
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Raymond_Ndibe
	Mar 7 2025, 9:00 PM

Description

The job contjob has two replicas as seen below:

Job name:	contjob
Command:	./contjob.sh
Job type:	continuous
Image:	bookworm
Port:	none
File log:	no
Output log:
Error log:
Emails:	none
Resources:	default
Replicas:	2
Mounts:	all
Retry:	no
Timeout:	no
Health check:	none
Status:	Running
Hints:	Last run at 2025-03-07T18:28:15Z. Pod in 'Running' phase. State
	'running'. Started at '2025-03-07T18:28:17Z'.

But when you run toolforge jobs logs contjob -f, we get only the logs from a single pod as seen below:

2025-03-07T20:48:48+00:00 [contjob-6dd45f87d-srb62] Fri Mar  7 08:48:48 PM UTC 2025
2025-03-07T20:48:48+00:00 [contjob-6dd45f87d-srb62] another loop!
2025-03-07T20:49:08+00:00 [contjob-6dd45f87d-srb62] Fri Mar  7 08:49:08 PM UTC 2025
2025-03-07T20:49:08+00:00 [contjob-6dd45f87d-srb62] another loop!
2025-03-07T20:49:28+00:00 [contjob-6dd45f87d-srb62] Fri Mar  7 08:49:28 PM UTC 2025
2025-03-07T20:49:28+00:00 [contjob-6dd45f87d-srb62] another loop!
2025-03-07T20:49:48+00:00 [contjob-6dd45f87d-srb62] Fri Mar  7 08:49:48 PM UTC 2025
2025-03-07T20:49:48+00:00 [contjob-6dd45f87d-srb62] another loop!
2025-03-07T20:50:08+00:00 [contjob-6dd45f87d-srb62] Fri Mar  7 08:50:08 PM UTC 2025
2025-03-07T20:50:08+00:00 [contjob-6dd45f87d-srb62] another loop!

This should be fixed to resemble the form below (here we are getting the logs from all containers in all target pods):

2025-03-07T20:39:59+00:00 [contjob-6dd45f87d-wb8cl] Fri Mar  7 08:39:59 PM UTC 2025
2025-03-07T20:39:59+00:00 [contjob-6dd45f87d-wb8cl] another loop!
2025-03-07T20:40:01+00:00 [contjob-6dd45f87d-wb8cl] container-2-for-pod
2025-03-07T20:40:01+00:00 [contjob-6dd45f87d-srb62] container-2-for-pod
2025-03-07T20:40:06+00:00 [contjob-6dd45f87d-srb62] Fri Mar  7 08:40:06 PM UTC 2025
2025-03-07T20:40:06+00:00 [contjob-6dd45f87d-srb62] another loop!
2025-03-07T20:40:11+00:00 [contjob-6dd45f87d-srb62] container-2-for-pod
2025-03-07T20:40:11+00:00 [contjob-6dd45f87d-wb8cl] container-2-for-pod
2025-03-07T20:40:19+00:00 [contjob-6dd45f87d-wb8cl] Fri Mar  7 08:40:19 PM UTC 2025
2025-03-07T20:40:19+00:00 [contjob-6dd45f87d-wb8cl] another loop!
2025-03-07T20:40:21+00:00 [contjob-6dd45f87d-srb62] container-2-for-pod
2025-03-07T20:40:21+00:00 [contjob-6dd45f87d-wb8cl] container-2-for-pod

Related Objects

Mentioned In: T398647: Move Kubernetes log source multi-pod handling from jobs-api to toolforge-weld

Event Timeline

Raymond_Ndibe created this task.Mar 7 2025, 9:00 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 7 2025, 9:00 PM

Raymond_Ndibe renamed this task from [jobs-api] toolforge jobs logs <name> -f should get the logs of all containers in all target pods to [jobs-api] `toolforge jobs logs <name> -f` should get the logs of all containers in all target pods.Mar 7 2025, 9:01 PM

Raymond_Ndibe renamed this task from [jobs-api] `toolforge jobs logs <name> -f` should get the logs of all containers in all target pods to [jobs-api] "toolforge jobs logs <name> -f" should get the logs of all containers in all target pods.

raymond-ndibe updated https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/148

[jobs-api] stream logs from all containers in all pods

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/77

[toolforge-weld] get all logs from all containers in all pods

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/87

[jobs-cli] include container info in log

Raymond_Ndibe changed the task status from Open to In Progress.Mar 7 2025, 9:37 PM

Raymond_Ndibe moved this task from Next Up to In Review on the Toolforge (Toolforge iteration 18) board.

dcaro triaged this task as Medium priority.Mar 11 2025, 2:48 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/720

[toolforge-deploy] increase milhistbot harbor quota to 2Gi

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/720