Page MenuHomePhabricator
Create Task
Maniphest T386480

[o11y,logging,infra] Deploy Loki to store Toolforge tool log data
Closed, ResolvedPublic
Actions

Description

Log Toolforge tool output to a Loki instance.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
P:toolforge::prometheus: Add scrape rules for Loki/Alloyoperations/puppetproduction+12 -0
toolforge: Add cookbook to mirror Loki-related imagescloud/wmcs-cookbooksmain+117 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• BstormT126083 overhaul labstore setup [tracking]
 Resolved• GTirloniT216988 labstore1004 - DISK CRITICAL - free space: /srv/tools 115904 MB (1% inode=79%):
 Resolved• BstormT217993 2019-03-10: tools and NFS share cleanup (high usage)
 Resolved• BstormT122508 Prevent overly-large log files
 DeclinedNoneT286847 Add webservice flag to mount project directory read-only
 OpentaaviT127367 [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools
 ResolvedFeaturetaaviT400913 [jobs-api] Allow querying logs for non-existent jobs
 OpenNoneT400914 [jobs-api] Remove file logging support
 OpenNoneT400915 [jobs-api] Disable file logging by default
 ResolvedtaaviT400916 [jobs-api] Support following logs from Loki
 ResolvedtaaviT401113 [jobs-api] Migrate to FastAPI
 OpenFeatureRaymond_NdibeT400917 [jobs-api] Allow customizing time to request Loki logs for
 ResolvedtaaviT398645 [jobs-api] Jobs API should query logs from Loki
 OpenNoneT400981 Check alerting for Loki
 OpenFeatureNoneT97861 [toolforge,infra] Centralized logging for Toolforge infrastructure logs
 ResolvedtaaviT386480 [o11y,logging,infra] Deploy Loki to store Toolforge tool log data
 ResolvedtaaviT396574 Provision object storage volumes for Loki
 ResolvedAndrewT396594 Create OpenStack role that allows object storage access only
 ResolveddcaroT397339 Request creation of toolsbeta-logging VPS project
 ResolvedNoneT397446 Request creation of tools-logging VPS project
 ResolvedAndrewT398447 Increase tools-logging object storage quotas

Event Timeline

• rook created this task.Feb 14 2025, 3:42 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 14 2025, 3:42 PM
• rook added subscribers: dcaro, Raymond_Ndibe, • aborrero, Andrew.Feb 14 2025, 8:37 PM

I believe
https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/tree/T386480?ref_type=heads
and
https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/tree/T386480?ref_type=heads

Will get loki installed in lima-kilo. @aborrero @dcaro @Andrew @Raymond_Ndibe opinions?

I fussed a little with the git clone in roles/k8s/tasks/toolforge-deploy_components.yaml to pull T386480 as it isn't merged yet, so I don't really have a full test of the patch. Is there a way to do a true test of a patch like this that I'm not realizing?

Thanks!

dcaro added a comment.Feb 18 2025, 9:47 AM

I fussed a little with the git clone in roles/k8s/tasks/toolforge-deploy_components.yaml to pull T386480 as it isn't merged yet, so I don't really have a full test of the patch. Is there a way to do a true test of a patch like this that I'm not realizing?

Depends on your definition of true test :)

For a full lima-kilo rebuild with the new code, you can pass the value lima_kilo_toolforge_deploy_repo.ref (defined under lima-klio roles/k8s/defaults/main.yaml), either editing the file or passing --extra-vars lima_kilo_toolforge_deploy_repo.ref=new/value when running start-devenv.sh.

If you just want to try the toolforge-deploy part of it you can apply the patch on the clone inside lima-kilo (under ~/toolforge-deploy) and run the deploy script ~/toolforge-deploy/deploy.sh by hand.

If you want to test also the lima-kilo changes without rebuilding, you can re-run ansible from within lima-kilo with toolforge_ansible_run.sh -t k8s (or fulling without params)

dcaro added a comment.Feb 18 2025, 10:26 AM

Will get loki installed in lima-kilo. @aborrero @dcaro @Andrew @Raymond_Ndibe opinions?

The patches look ok to me (have not tested them yet, testing something else...), they have no extra config for loki yet right?

• rook added a comment.Feb 18 2025, 11:47 AM
In T386480#10558944, @dcaro wrote:

The patches look ok to me (have not tested them yet, testing something else...), they have no extra config for loki yet right?

Correct this is installing a stock loki. Do we think we should merge these and continue from here or take a different approach?

CodeReviewBot added a project: Patch-For-Review.Feb 18 2025, 3:51 PM

rook opened https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/226

Adding loki

CodeReviewBot added a comment.Feb 18 2025, 3:51 PM

rook opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/669

Adding loki to install

dcaro added a comment.Feb 19 2025, 8:46 AM

Do we think we should merge these and continue from here or take a different approach?

It should be ok, we could not install loki by default too and avoid people that don't need it yet from getting it installed, depends on how soon you expect to start integrating it with the rest of toolforge.

dcaro renamed this task from loki into lima-kilo to [o11y,logging,infra] loki into lima-kilo.Feb 19 2025, 8:51 AM
dcaro triaged this task as High priority.
dcaro edited projects, added Toolforge (Toolforge iteration 17); removed Toolforge.
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 17) board.
• aborrero mentioned this in T127367: [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools.Feb 19 2025, 9:36 AM
• aborrero moved this task from Backlog to Radar/observer on the User-aborrero board.Feb 19 2025, 1:25 PM
• rook added a comment.Feb 19 2025, 4:40 PM

As I consider it more I guess it doesn't make much of a difference if it is merged now, as about the biggest "risk" of not merging is that I eventually push a toolforge-deploy_components.yaml that is set to pull T386480. So perhaps it is better to wait until we decide more exactly what we want before merging.

dcaro edited projects, added Toolforge (Toolforge iteration 18); removed Toolforge (Toolforge iteration 17).Feb 25 2025, 3:29 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 18) board.
dcaro removed • rook as the assignee of this task.Mar 18 2025, 3:40 PM
dcaro moved this task from In Progress to Next Up on the Toolforge (Toolforge iteration 18) board.
dcaro edited projects, added Toolforge; removed Toolforge (Toolforge iteration 18).Mar 18 2025, 3:47 PM
dcaro moved this task from Backlog to Ready to be worked on on the Toolforge board.May 14 2025, 5:09 PM
CodeReviewBot added a comment.May 23 2025, 9:31 AM

taavi closed https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/669

Adding loki to install

taavi renamed this task from [o11y,logging,infra] loki into lima-kilo to [o11y,logging,infra] Deploy Loki to store Toolforge log data.May 23 2025, 2:32 PM
taavi removed projects: User-aborrero, Epic.
taavi removed a subscriber: • rook.
taavi added a parent task: T97861: [toolforge,infra] Centralized logging for Toolforge infrastructure logs.May 23 2025, 2:36 PM
CodeReviewBot added a comment.May 23 2025, 2:54 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/796

logging: Init component

taavi claimed this task.May 23 2025, 2:54 PM
taavi subscribed.

Tentatively claiming.

CodeReviewBot added a comment.Jun 6 2025, 12:43 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/807

logging: Add basic rate limiting and retention config

gerritbot added a comment.Jun 9 2025, 12:23 PM

Change #1154802 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/wmcs-cookbooks@main] toolforge: Add cookbook to mirror Loki-related images

https://gerrit.wikimedia.org/r/1154802

gerritbot added a comment.Jun 11 2025, 11:06 AM

Change #1154802 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] toolforge: Add cookbook to mirror Loki-related images

https://gerrit.wikimedia.org/r/1154802

taavi mentioned this in rCCKBe38be2f8bd4a: toolforge: Add cookbook to mirror Loki-related images.Jun 11 2025, 11:07 AM
CodeReviewBot added a comment.Jun 11 2025, 12:02 PM

taavi closed https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/226

Adding loki

taavi lowered the priority of this task from High to Medium.Jun 13 2025, 9:31 AM
CodeReviewBot added a comment.Jun 17 2025, 4:33 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/796

logging: Init component

CodeReviewBot added a comment.Jun 17 2025, 4:33 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/807

logging: Add basic rate limiting and retention config

Maintenance_bot removed a project: Patch-For-Review.Jun 17 2025, 5:31 PM
CodeReviewBot added a comment.Jun 23 2025, 5:15 PM

taavi updated https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/821

logging: Add values to deploy to toolsbeta

taavi closed subtask T396574: Provision object storage volumes for Loki as Resolved.Jun 23 2025, 5:15 PM
taavi mentioned this in T397651: CAS not letting new Toolsbeta-logging developer account log in.Jun 23 2025, 5:19 PM
CodeReviewBot added a comment.Jun 24 2025, 9:56 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/821

logging: Add values to deploy to toolsbeta

CodeReviewBot added a project: Patch-For-Review.Jun 24 2025, 10:00 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/823

logging: Fix path to get_secret.sh

CodeReviewBot added a comment.Jun 24 2025, 10:04 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/824

logging: loki: Add missing emptyDir mounts in toolsbeta

CodeReviewBot added a comment.Jun 24 2025, 10:16 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/825

logging: loki: Set nameOverride

CodeReviewBot added a comment.Jun 24 2025, 10:26 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/827

logging: loki: Add network policy rule for object storage access

CodeReviewBot added a comment.Jun 24 2025, 10:26 AM

taavi updated https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/826

logging: alloy: Fix loki write service name

CodeReviewBot added a comment.Jun 24 2025, 11:53 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/55

logging: Deploy remaining Loki buckets

CodeReviewBot added a comment.Jun 25 2025, 8:06 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/834

logging: loki: Add second Loki instance for infrastructure logs

CodeReviewBot added a comment.Jun 25 2025, 8:06 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/835

logging: alloy: Add routing for infrastructure logs

gerritbot added a comment.Jun 25 2025, 10:22 AM

Change #1163729 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::prometheus: Add scrape rules for Loki/Alloy

https://gerrit.wikimedia.org/r/1163729

CodeReviewBot added a comment.Jun 25 2025, 5:25 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/823

logging: Fix path to get_secret.sh

CodeReviewBot added a comment.Jun 25 2025, 5:26 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/824

logging: loki: Add missing emptyDir mounts in toolsbeta

CodeReviewBot added a comment.Jun 26 2025, 9:38 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/825

logging: loki: Set nameOverride

CodeReviewBot added a comment.Jun 26 2025, 9:38 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/826

logging: alloy: Fix loki write service name

taavi mentioned this in T398285: Decision request - Reuse toolforge user tools central logging for toolforge infrastructure logging.Jul 1 2025, 10:06 AM
CodeReviewBot added a comment.Jul 1 2025, 11:42 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/827

logging: loki: Add network policy rule for object storage access

CodeReviewBot added a comment.Jul 1 2025, 11:43 AM

taavi updated https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/836

logging: alloy: Allow running on the entire cluster

CodeReviewBot added a comment.Jul 1 2025, 12:08 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/856

logging: Deploy to tools

taavi renamed this task from [o11y,logging,infra] Deploy Loki to store Toolforge log data to [o11y,logging,infra] Deploy Loki to store Toolforge tool log data.Jul 1 2025, 12:16 PM
taavi updated the task description. (Show Details)
CodeReviewBot added a comment.Jul 1 2025, 2:09 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/856

logging: Deploy to tools

gerritbot added a comment.Jul 1 2025, 2:21 PM

Change #1163729 merged by Majavah:

[operations/puppet@production] P:toolforge::prometheus: Add scrape rules for Loki/Alloy

https://gerrit.wikimedia.org/r/1163729

CodeReviewBot added a comment.Jul 1 2025, 3:09 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/858

logging: alloy: Deploy to two more tools workers

CodeReviewBot added a comment.Jul 1 2025, 3:15 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/858

logging: alloy: Deploy to two more tools workers

CodeReviewBot added a comment.Jul 2 2025, 12:07 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/253

Deploy logging stack by default

taavi added a subtask: T398447: Increase tools-logging object storage quotas.Jul 2 2025, 1:51 PM
Andrew closed subtask T398447: Increase tools-logging object storage quotas as Resolved.Jul 2 2025, 5:44 PM
CodeReviewBot added a comment.Jul 3 2025, 8:21 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/862

logging: alloy: Deploy to more workers

CodeReviewBot added a comment.Jul 3 2025, 8:25 AM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/862

logging: alloy: Deploy to more workers

CodeReviewBot added a comment.Jul 3 2025, 12:22 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/864

logging: alloy: Deploy to even more nodes

CodeReviewBot added a comment.Jul 3 2025, 1:23 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/864

logging: alloy: Deploy to even more nodes

taavi added a parent task: T398645: [jobs-api] Jobs API should query logs from Loki.Jul 3 2025, 3:53 PM
CodeReviewBot added a comment.Jul 7 2025, 7:54 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/869

logging: alloy: Deploy to double the workers

CodeReviewBot added a comment.Jul 7 2025, 8:18 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/869

logging: alloy: Deploy to double the workers

CodeReviewBot added a comment.Jul 7 2025, 2:17 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/253

Deploy logging stack by default

CodeReviewBot added a comment.Jul 8 2025, 10:04 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/871

logging: alloy: Deploy cluster-wide on tools

CodeReviewBot added a comment.Jul 8 2025, 12:37 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/871

logging: alloy: Deploy cluster-wide on tools

taavi closed this task as Resolved.Jul 8 2025, 5:44 PM
taavi added a parent task: T400981: Check alerting for Loki.Aug 1 2025, 9:36 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits