Provision object storage volumes for Loki
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	taavi
	Jun 11 2025, 9:00 AM

Description

I'd like to deploy Loki to collect Toolsbeta tool and infrastructure logs next. It will persist data in our object storage service.

This task is to figure out bucket provisioning, setting some initial per-bucket limits, and credentials for the Loki service (in K8s) to talk to the object storage service. Preferrably the bucket config would happen via Tofu.

upstream docs: https://grafana.com/docs/loki/latest/configure/storage/

Details

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
logging: Use separate app creds for separate projects	repos/cloud/toolforge/tofu-provisioning!52	taavi	main-Ib3fcc96e3ede553ef003d9062513016b1b2511ed	main
Revert "shared: Provision storage buckets for Loki"	repos/cloud/toolforge/tofu-provisioning!51	taavi	main-I99a5f3cb7f00f65a9f10a03ee1e082fdc7cfafb8	main
Provision log storage buckets in a separate project	repos/cloud/toolforge/tofu-provisioning!50	taavi	main-I1384635fe4afe95f8ba034f20fe8ce61efe1ff4b	main
shared: Provision storage buckets for Loki	repos/cloud/toolforge/tofu-provisioning!49	taavi	main-I2123be8109ae73e3e9944cb69fbd7a00f596f99b	main

Customize query in GitLab

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		• Bstorm	T126083 overhaul labstore setup [tracking]
Resolved		• GTirloni	T216988 labstore1004 - DISK CRITICAL - free space: /srv/tools 115904 MB (1% inode=79%):
Resolved		• Bstorm	T217993 2019-03-10: tools and NFS share cleanup (high usage)
Resolved		• Bstorm	T122508 Prevent overly-large log files
Declined		None	T286847 Add webservice flag to mount project directory read-only
Open		taavi	T127367 [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools
Resolved	Feature	taavi	T400913 [jobs-api] Allow querying logs for non-existent jobs
Open		None	T400914 [jobs-api] Remove file logging support
Open		None	T400915 [jobs-api] Disable file logging by default
Resolved		taavi	T400916 [jobs-api] Support following logs from Loki
Resolved		taavi	T401113 [jobs-api] Migrate to FastAPI
Open	Feature	Raymond_Ndibe	T400917 [jobs-api] Allow customizing time to request Loki logs for
Resolved		taavi	T398645 [jobs-api] Jobs API should query logs from Loki
Open		None	T400981 Check alerting for Loki
Open	Feature	None	T97861 [toolforge,infra] Centralized logging for Toolforge infrastructure logs
Resolved		taavi	T386480 [o11y,logging,infra] Deploy Loki to store Toolforge tool log data
Resolved		taavi	T396574 Provision object storage volumes for Loki
Resolved		Andrew	T396594 Create OpenStack role that allows object storage access only
Resolved		dcaro	T397339 Request creation of toolsbeta-logging VPS project
Resolved		None	T397446 Request creation of tools-logging VPS project

Event Timeline

taavi created this task.Jun 11 2025, 9:00 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 11 2025, 9:00 AM

taavi triaged this task as High priority.Jun 11 2025, 9:30 AM

taavi updated the task description. (Show Details)Jun 11 2025, 9:32 AM

@Andrew do you happen to know if individual buckets can be configured to have a storage limit applied to them without having to go through the Rados admin CLI?

There is an API as well as a CLI:

https://docs.ceph.com/en/latest/radosgw/adminops/

So it should be possible, although I have never tried to use it.

I'm still a bit attached to the idea of mapping tools to keystone projects, at which point users could have a shared quota used for tool-specific objects or logs (and decide if they love their logs or their media more.) but that may be way out of scope for what you're planning.

taavi lowered the priority of this task from High to Medium.Jun 13 2025, 9:31 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/49

shared: Provision storage buckets for Loki

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/49

shared: Provision storage buckets for Loki

Maintenance_bot removed a project: Patch-For-Review.Jun 18 2025, 12:31 PM

taavi mentioned this in T397339: Request creation of toolsbeta-logging VPS project.Jun 18 2025, 2:00 PM

taavi added a subtask: T397339: Request creation of toolsbeta-logging VPS project.

taavi closed subtask T396594: Create OpenStack role that allows object storage access only as Resolved.Jun 18 2025, 2:04 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/50

Use separate project for log storage buckets

taavi closed subtask T397339: Request creation of toolsbeta-logging VPS project as Resolved.Jun 19 2025, 12:08 PM