Page MenuHomePhabricator
Create Task
Maniphest T386480

[o11y,logging,infra] Deploy Loki to store Toolforge tool log data
Closed, ResolvedPublic
Actions

Description

Log Toolforge tool output to a Loki instance.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
P:toolforge::prometheus: Add scrape rules for Loki/Alloyoperations/puppetproduction+12 -0
toolforge: Add cookbook to mirror Loki-related imagescloud/wmcs-cookbooksmain+117 -0
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
logging: alloy: Deploy cluster-wide on toolsrepos/cloud/toolforge/toolforge-deploy!871taavimain-I7a4ff905e57696ed6534226596428a9294a1e3f6main
logging: alloy: Deploy to double the workersrepos/cloud/toolforge/toolforge-deploy!869taavimain-I3094dc7db753f36154b35cc9c9b3b693a23bc4dcmain
logging: alloy: Deploy to even more nodesrepos/cloud/toolforge/toolforge-deploy!864taavimain-Ifb17bf62ef6f5cf8a6e186c692ce6a456b163b3bmain
logging: alloy: Deploy to more workersrepos/cloud/toolforge/toolforge-deploy!862taavimain-I395644a8e25e1cdef253a09d25320464fa696d6emain
Deploy logging stack by defaultrepos/cloud/toolforge/lima-kilo!253taavimain-I72ab6ba788ca68e9b86bf3bbdb338b15c913eaeemain
logging: alloy: Deploy to two more tools workersrepos/cloud/toolforge/toolforge-deploy!858taavimain-I0b5a86931095bf9a951e14c9b38540df97e7f0dfmain
logging: Deploy to toolsrepos/cloud/toolforge/toolforge-deploy!856taavimain-I2a75f16c20f737eaae579a36b0a927a037469c0fmain
logging: alloy: Allow running on the entire clusterrepos/cloud/toolforge/toolforge-deploy!836taavimain-I7c5569f698935ca9794d6455fb76ea376d1f9b83main-I156a2539de7b40a8aeeed3933fe37d01c93d17a5
logging: alloy: Add routing for infrastructure logsrepos/cloud/toolforge/toolforge-deploy!835taavimain-I156a2539de7b40a8aeeed3933fe37d01c93d17a5main-Ie19e6900e99bd48acbbcf8b86491318b614e8abe
logging: loki: Add second Loki instance for infrastructure logsrepos/cloud/toolforge/toolforge-deploy!834taavimain-Ie19e6900e99bd48acbbcf8b86491318b614e8abemain
logging: loki: Add network policy rule for object storage accessrepos/cloud/toolforge/toolforge-deploy!827taavimain-I729571e8337cbbf6b433eb5d6180b7dc6d3e4bb7main
logging: alloy: Fix loki write service namerepos/cloud/toolforge/toolforge-deploy!826taavimain-I7a15f3e0883a50d8c4cb808fe6f055f09b446c1bmain
logging: loki: Set nameOverriderepos/cloud/toolforge/toolforge-deploy!825taavimain-I59df6ccd608477f2810572f634ad259d9d9c0320main
logging: loki: Add missing emptyDir mounts in toolsbetarepos/cloud/toolforge/toolforge-deploy!824taavimain-I3639ecc2f5ab9b2b4de99d3e2ce13878ea98f62emain
logging: Fix path to get_secret.shrepos/cloud/toolforge/toolforge-deploy!823taavimain-Ia5b1c14eb31da968d2010289033c6b5b53b07311main
logging: Add values to deploy to toolsbetarepos/cloud/toolforge/toolforge-deploy!821taavimain-I054208e27e5db1902a411d3b8df9a317c7aa5488main
logging: Add basic rate limiting and retention configrepos/cloud/toolforge/toolforge-deploy!807taavimain-Ieb3604c5872169a2e8cd173b52177f46a0ef13femain
logging: Init componentrepos/cloud/toolforge/toolforge-deploy!796taavimain-If8f503514316703ce91f966fb6ad40b04ef8fdd0main
Adding loki to installrepos/cloud/toolforge/toolforge-deploy!669rookT386480main
Adding lokirepos/cloud/toolforge/lima-kilo!226rookT386480main
Show related patches Customize query in GitLab

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• BstormT126083 overhaul labstore setup [tracking]
 Resolved• GTirloniT216988 labstore1004 - DISK CRITICAL - free space: /srv/tools 115904 MB (1% inode=79%):
 Resolved• BstormT217993 2019-03-10: tools and NFS share cleanup (high usage)
 Resolved• BstormT122508 Prevent overly-large log files
 DeclinedNoneT286847 Add webservice flag to mount project directory read-only
 OpentaaviT127367 [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools
 ResolvedFeaturetaaviT400913 [jobs-api] Allow querying logs for non-existent jobs
 OpenNoneT400914 [jobs-api] Remove file logging support
 OpenNoneT400915 [jobs-api] Disable file logging by default
 ResolvedtaaviT400916 [jobs-api] Support following logs from Loki
 ResolvedtaaviT401113 [jobs-api] Migrate to FastAPI
 OpenFeatureRaymond_NdibeT400917 [jobs-api] Allow customizing time to request Loki logs for
 ResolvedtaaviT398645 [jobs-api] Jobs API should query logs from Loki
 OpenNoneT400981 Check alerting for Loki
 OpenFeatureNoneT97861 [toolforge,infra] Centralized logging for Toolforge infrastructure logs
 ResolvedtaaviT386480 [o11y,logging,infra] Deploy Loki to store Toolforge tool log data
 ResolvedtaaviT396574 Provision object storage volumes for Loki
 ResolvedAndrewT396594 Create OpenStack role that allows object storage access only
 ResolveddcaroT397339 Request creation of toolsbeta-logging VPS project
 ResolvedNoneT397446 Request creation of tools-logging VPS project
 ResolvedAndrewT398447 Increase tools-logging object storage quotas

Event Timeline

• rook created this task.Feb 14 2025, 3:42 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 14 2025, 3:42 PM
• rook added subscribers: dcaro, Raymond_Ndibe, • aborrero, Andrew.Feb 14 2025, 8:37 PM

I believe
https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/tree/T386480?ref_type=heads
and
https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/tree/T386480?ref_type=heads

Will get loki installed in lima-kilo. @aborrero @dcaro @Andrew @Raymond_Ndibe opinions?

I fussed a little with the git clone in roles/k8s/tasks/toolforge-deploy_components.yaml to pull T386480 as it isn't merged yet, so I don't really have a full test of the patch. Is there a way to do a true test of a patch like this that I'm not realizing?

Thanks!

dcaro added a comment.Feb 18 2025, 9:47 AM

I fussed a little with the git clone in roles/k8s/tasks/toolforge-deploy_components.yaml to pull T386480 as it isn't merged yet, so I don't really have a full test of the patch. Is there a way to do a true test of a patch like this that I'm not realizing?

Depends on your definition of true test :)

For a full lima-kilo rebuild with the new code, you can pass the value lima_kilo_toolforge_deploy_repo.ref (defined under lima-klio roles/k8s/defaults/main.yaml), either editing the file or passing --extra-vars lima_kilo_toolforge_deploy_repo.ref=new/value when running start-devenv.sh.

If you just want to try the toolforge-deploy part of it you can apply the patch on the clone inside lima-kilo (under ~/toolforge-deploy) and run the deploy script ~/toolforge-deploy/deploy.sh by hand.

If you want to test also the lima-kilo changes without rebuilding, you can re-run ansible from within lima-kilo with toolforge_ansible_run.sh -t k8s (or fulling without params)

dcaro added a comment.Feb 18 2025, 10:26 AM

Will get loki installed in lima-kilo. @aborrero @dcaro @Andrew @Raymond_Ndibe opinions?

The patches look ok to me (have not tested them yet, testing something else...), they have no extra config for loki yet right?

• rook added a comment.Feb 18 2025, 11:47 AM
In T386480#10558944, @dcaro wrote:

The patches look ok to me (have not tested them yet, testing something else...), they have no extra config for loki yet right?

Correct this is installing a stock loki. Do we think we should merge these and continue from here or take a different approach?

CodeReviewBot added a project: Patch-For-Review.Feb 18 2025, 3:51 PM

rook opened https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/226

Adding loki

CodeReviewBot added a comment.Feb 18 2025, 3:51 PM

rook opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/669

Adding loki to install

dcaro added a comment.Feb 19 2025, 8:46 AM

Do we think we should merge these and continue from here or take a different approach?

It should be ok, we could not install loki by default too and avoid people that don't need it yet from getting it installed, depends on how soon you expect to start integrating it with the rest of toolforge.

dcaro renamed this task from loki into lima-kilo to [o11y,logging,infra] loki into lima-kilo.Feb 19 2025, 8:51 AM
dcaro triaged this task as High priority.
dcaro edited projects, added Toolforge (Toolforge iteration 17); removed Toolforge.
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 17) board.
• aborrero mentioned this in T127367: [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools.Feb 19 2025, 9:36 AM
• aborrero moved this task from Backlog to Radar/observer on the User-aborrero board.Feb 19 2025, 1:25 PM
• rook added a comment.Feb 19 2025, 4:40 PM

As I consider it more I guess it doesn't make much of a difference if it is merged now, as about the biggest "risk" of not merging is that I eventually push a toolforge-deploy_components.yaml that is set to pull T386480. So perhaps it is better to wait until we decide more exactly what we want before merging.

dcaro edited projects, added Toolforge (Toolforge iteration 18); removed Toolforge (Toolforge iteration 17).Feb 25 2025, 3:29 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 18) board.
dcaro removed • rook as the assignee of this task.Mar 18 2025, 3:40 PM
dcaro moved this task from In Progress to Next Up on the Toolforge (Toolforge iteration 18) board.
dcaro edited projects, added Toolforge; removed Toolforge (Toolforge iteration 18).Mar 18 2025, 3:47 PM
dcaro moved this task from Backlog to Ready to be worked on on the Toolforge board.May 14 2025, 5:09 PM
CodeReviewBot added a comment.May 23 2025, 9:31 AM

taavi closed https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/669

Adding loki to install

taavi renamed this task from [o11y,logging,infra] loki into lima-kilo to [o11y,logging,infra] Deploy Loki to store Toolforge log data.May 23 2025, 2:32 PM
taavi removed projects: User-aborrero, Epic.
taavi removed a subscriber: • rook.
taavi added a parent task: T97861: [toolforge,infra] Centralized logging for Toolforge infrastructure logs.May 23 2025, 2:36 PM
CodeReviewBot added a comment.May 23 2025, 2:54 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/796

logging: Init component

taavi claimed this task.May 23 2025, 2:54 PM
taavi subscribed.

Tentatively claiming.

CodeReviewBot added a comment.Jun 6 2025, 12:43 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/807

logging: Add basic rate limiting and retention config

gerritbot added a comment.Jun 9 2025, 12:23 PM

Change #1154802 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/wmcs-cookbooks@main] toolforge: Add cookbook to mirror Loki-related images

https://gerrit.wikimedia.org/r/1154802

gerritbot added a comment.Jun 11 2025, 11:06 AM

Change #1154802 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] toolforge: Add cookbook to mirror Loki-related images

https://gerrit.wikimedia.org/r/1154802

taavi mentioned this in rCCKBe38be2f8bd4a: toolforge: Add cookbook to mirror Loki-related images.Jun 11 2025, 11:07 AM
CodeReviewBot added a comment.Jun 11 2025, 12:02 PM

taavi closed https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/226

Adding loki

taavi lowered the priority of this task from High to Medium.Jun 13 2025, 9:31 AM
CodeReviewBot added a comment.Jun 17 2025, 4:33 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/796

logging: Init component

CodeReviewBot added a comment.Jun 17 2025, 4:33 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/807

logging: Add basic rate limiting and retention config

Maintenance_bot removed a project: Patch-For-Review.Jun 17 2025, 5:31 PM
CodeReviewBot added a comment.Jun 23 2025, 5:15 PM

taavi updated https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/821

logging: Add values to deploy to toolsbeta

taavi closed subtask T396574: Provision object storage volumes for Loki as Resolved.Jun 23 2025, 5:15 PM
taavi mentioned this in T397651: CAS not letting new Toolsbeta-logging developer account log in.Jun 23 2025, 5:19 PM
CodeReviewBot added a comment.Jun 24 2025, 9:56 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/821

logging: Add values to deploy to toolsbeta

CodeReviewBot added a project: Patch-For-Review.Jun 24 2025, 10:00 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/823

logging: Fix path to get_secret.sh

CodeReviewBot added a comment.Jun 24 2025, 10:04 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/824

logging: loki: Add missing emptyDir mounts in toolsbeta

CodeReviewBot added a comment.Jun 24 2025, 10:16 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/825

logging: loki: Set nameOverride

CodeReviewBot added a comment.Jun 24 2025, 10:26 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/827

logging: loki: Add network policy rule for object storage access

CodeReviewBot added a comment.Jun 24 2025, 10:26 AM

taavi updated https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/826

logging: alloy: Fix loki write service name

CodeReviewBot added a comment.Jun 24 2025, 11:53 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/55

logging: Deploy remaining Loki buckets

CodeReviewBot added a comment.Jun 25 2025, 8:06 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/834

logging: loki: Add second Loki instance for infrastructure logs

CodeReviewBot added a comment.Jun 25 2025, 8:06 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/835

logging: alloy: Add routing for infrastructure logs

gerritbot added a comment.Jun 25 2025, 10:22 AM

Change #1163729 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::prometheus: Add scrape rules for Loki/Alloy

https://gerrit.wikimedia.org/r/1163729

CodeReviewBot added a comment.Jun 25 2025, 5:25 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/823

logging: Fix path to get_secret.sh

CodeReviewBot added a comment.Jun 25 2025, 5:26 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/824

logging: loki: Add missing emptyDir mounts in toolsbeta

CodeReviewBot added a comment.Jun 26 2025, 9:38 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/825

logging: loki: Set nameOverride

CodeReviewBot added a comment.Jun 26 2025, 9:38 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/826

logging: alloy: Fix loki write service name

taavi mentioned this in T398285: Decision request - Reuse toolforge user tools central logging for toolforge infrastructure logging.Jul 1 2025, 10:06 AM
CodeReviewBot added a comment.Jul 1 2025, 11:42 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/827

logging: loki: Add network policy rule for object storage access

CodeReviewBot added a comment.Jul 1 2025, 11:43 AM

taavi updated https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/836

logging: alloy: Allow running on the entire cluster

CodeReviewBot added a comment.Jul 1 2025, 12:08 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/856

logging: Deploy to tools

taavi renamed this task from [o11y,logging,infra] Deploy Loki to store Toolforge log data to [o11y,logging,infra] Deploy Loki to store Toolforge tool log data.Jul 1 2025, 12:16 PM
taavi updated the task description. (Show Details)
CodeReviewBot added a comment.Jul 1 2025, 2:09 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/856

logging: Deploy to tools

gerritbot added a comment.Jul 1 2025, 2:21 PM

Change #1163729 merged by Majavah:

[operations/puppet@production] P:toolforge::prometheus: Add scrape rules for Loki/Alloy

https://gerrit.wikimedia.org/r/1163729

CodeReviewBot added a comment.Jul 1 2025, 3:09 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/858

logging: alloy: Deploy to two more tools workers

CodeReviewBot added a comment.Jul 1 2025, 3:15 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/858

logging: alloy: Deploy to two more tools workers

CodeReviewBot added a comment.Jul 2 2025, 12:07 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/253

Deploy logging stack by default

taavi added a subtask: T398447: Increase tools-logging object storage quotas.Jul 2 2025, 1:51 PM
Andrew closed subtask T398447: Increase tools-logging object storage quotas as Resolved.Jul 2 2025, 5:44 PM
CodeReviewBot added a comment.Jul 3 2025, 8:21 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/862

logging: alloy: Deploy to more workers

CodeReviewBot added a comment.Jul 3 2025, 8:25 AM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/862

logging: alloy: Deploy to more workers

CodeReviewBot added a comment.Jul 3 2025, 12:22 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/864

logging: alloy: Deploy to even more nodes

CodeReviewBot added a comment.Jul 3 2025, 1:23 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/864

logging: alloy: Deploy to even more nodes

taavi added a parent task: T398645: [jobs-api] Jobs API should query logs from Loki.Jul 3 2025, 3:53 PM
CodeReviewBot added a comment.Jul 7 2025, 7:54 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/869

logging: alloy: Deploy to double the workers

CodeReviewBot added a comment.Jul 7 2025, 8:18 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/869

logging: alloy: Deploy to double the workers

CodeReviewBot added a comment.Jul 7 2025, 2:17 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/253

Deploy logging stack by default

CodeReviewBot added a comment.Jul 8 2025, 10:04 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/871

logging: alloy: Deploy cluster-wide on tools

CodeReviewBot added a comment.Jul 8 2025, 12:37 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/871

logging: alloy: Deploy cluster-wide on tools

taavi closed this task as Resolved.Jul 8 2025, 5:44 PM
taavi added a parent task: T400981: Check alerting for Loki.Aug 1 2025, 9:36 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits