Page MenuHomePhabricator
Create Task
Maniphest T398645

[jobs-api] Jobs API should query logs from Loki
Closed, ResolvedPublic
Actions

Description

The Jobs API needs to be changed to query logs from Loki instead of from the Kubernetes buffers directly.

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
logging: loki: Add network policy for jobs-api read accessrepos/cloud/toolforge/toolforge-deploy!868taavimain-I374440d2e64a758c1c663535b9784a362b1e9539main
Customize query in GitLab

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• BstormT126083 overhaul labstore setup [tracking]
 Resolved• GTirloniT216988 labstore1004 - DISK CRITICAL - free space: /srv/tools 115904 MB (1% inode=79%):
 Resolved• BstormT217993 2019-03-10: tools and NFS share cleanup (high usage)
 Resolved• BstormT122508 Prevent overly-large log files
 DeclinedNoneT286847 Add webservice flag to mount project directory read-only
 OpenNoneT127367 [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools
 ResolvedFeaturetaaviT400913 [jobs-api] Allow querying logs for non-existent jobs
 OpenNoneT400914 [jobs-api] Remove file logging support
 OpenNoneT400915 [jobs-api] Disable file logging by default
 ResolvedtaaviT400916 [jobs-api] Support following logs from Loki
 ResolvedFeatureRaymond_NdibeT400917 [jobs-api] Allow customizing time to request Loki logs for
 ResolvedtaaviT398645 [jobs-api] Jobs API should query logs from Loki
 ResolvedtaaviT386480 [o11y,logging,infra] Deploy Loki to store Toolforge tool log data
 ResolvedtaaviT396574 Provision object storage volumes for Loki
 ResolvedAndrewT396594 Create OpenStack role that allows object storage access only
 ResolveddcaroT397339 Request creation of toolsbeta-logging VPS project
 ResolvedNoneT397446 Request creation of tools-logging VPS project
 ResolvedAndrewT398447 Increase tools-logging object storage quotas
 ResolvedtaaviT398647 Move Kubernetes log source multi-pod handling from jobs-api to toolforge-weld
 ResolvedtaaviT400795 [jobs-api] 400 bad request when trying to load a large number of logs from Loki

Event Timeline

taavi created this task.Jul 3 2025, 3:49 PM
taavi triaged this task as High priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 3 2025, 3:49 PM
taavi lowered the priority of this task from High to Medium.Jul 3 2025, 3:49 PM
taavi updated the task description. (Show Details)
taavi claimed this task.Jul 3 2025, 3:52 PM
taavi added a subtask: T386480: [o11y,logging,infra] Deploy Loki to store Toolforge tool log data.
taavi mentioned this in T127367: [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools.Jul 3 2025, 3:56 PM
CodeReviewBot added a project: Patch-For-Review.Jul 4 2025, 12:31 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/180

Query logs from Loki

CodeReviewBot added a comment.Jul 7 2025, 7:48 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/868

logging: loki: Add network policy for jobs-api read access

CodeReviewBot added a comment.Jul 7 2025, 8:17 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/868

logging: loki: Add network policy for jobs-api read access

dcaro moved this task from Backlog to Toolforge iteration 21 on the Toolforge board.Jul 7 2025, 4:30 PM
dcaro edited projects, added: Toolforge (Toolforge iteration 21); removed: Toolforge.
taavi closed subtask T386480: [o11y,logging,infra] Deploy Loki to store Toolforge tool log data as Resolved.Jul 8 2025, 5:44 PM
dcaro edited projects, added: Toolforge (Toolforge iteration 22); removed: Toolforge (Toolforge iteration 21).Jul 14 2025, 9:49 AM
taavi changed the task status from Open to In Progress.Jul 23 2025, 10:31 AM
taavi moved this task from Next Up to In Review on the Toolforge (Toolforge iteration 22) board.
taavi closed subtask T398647: Move Kubernetes log source multi-pod handling from jobs-api to toolforge-weld as Resolved.Jul 29 2025, 9:40 AM
CodeReviewBot added a comment.Jul 30 2025, 8:09 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/180

Query logs from Loki

CodeReviewBot added a comment.Jul 30 2025, 8:12 AM

group_203_bot_f4d95069bb2675e4ce1fff090c1c1620 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/886

jobs-api: bump to 0.0.388-20250730081001-aabaef95

CodeReviewBot added a comment.Jul 30 2025, 8:33 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/886

jobs-api: bump to 0.0.388-20250730081001-aabaef95

CodeReviewBot added a comment.Jul 30 2025, 8:35 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/887

jobs-api: local: Add Loki URL setting

CodeReviewBot added a comment.Jul 30 2025, 8:45 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/888

jobs-api: toolsbeta: Add Loki URL setting

CodeReviewBot added a comment.Jul 30 2025, 8:49 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/887

jobs-api: local: Add Loki URL setting

CodeReviewBot added a comment.Jul 30 2025, 8:52 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/888

jobs-api: toolsbeta: Add Loki URL setting

Maintenance_bot removed a project: Patch-For-Review.Jul 30 2025, 9:31 AM
taavi changed the status of subtask T400795: [jobs-api] 400 bad request when trying to load a large number of logs from Loki from Open to In Progress.Jul 30 2025, 9:57 AM
taavi closed subtask T400795: [jobs-api] 400 bad request when trying to load a large number of logs from Loki as Resolved.Jul 31 2025, 3:07 PM
CodeReviewBot added a project: Patch-For-Review.Jul 31 2025, 3:39 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/891

jobs-api: tools: Add Loki URL setting

taavi added a parent task: T400913: [jobs-api] Allow querying logs for non-existent jobs.Jul 31 2025, 3:43 PM
taavi added a parent task: T400915: [jobs-api] Disable file logging by default.Jul 31 2025, 3:53 PM
taavi added a parent task: T400916: [jobs-api] Support following logs from Loki.
taavi closed this task as Resolved.Jul 31 2025, 3:59 PM
taavi added a parent task: T400917: [jobs-api] Allow customizing time to request Loki logs for.

This is live and I have filed a bunch of tasks to improve the basic implementation.

CodeReviewBot added a comment.Jul 31 2025, 4:02 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/891

jobs-api: tools: Add Loki URL setting

Maintenance_bot removed a project: Patch-For-Review.Jul 31 2025, 4:37 PM
dcaro moved this task from In Review to Done on the Toolforge (Toolforge iteration 22) board.Aug 5 2025, 8:36 AM
taavi mentioned this in T349775: [toolforge,jobs] "toolforge jobs logs" fails when job has not started yet.Aug 5 2025, 9:08 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits