Page MenuHomePhabricator
Create Task
Maniphest T431248

[toolsbeta,infra,logging] We are reaching 96% of quota by objects
Closed, ResolvedPublic
Actions

Description

From an alert in https://alerts.wikimedia.org/?q=team%3Dwmcs:

Object storage quota by 'objects' is 83.65% full for project toolsbeta-logging

Dashboard: https://grafana-rw.wikimedia.org/d/7120b794-4638-49f5-bccd-9716efc60f24/wmcs-object-storage-quotas?forceLogin=true&from=now-30m&orgId=1&timezone=utc&to=now

Runbook: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/ObjectStorageObjectQuotaFull

There was a big bump starting in late June, investigate also why if there's any cause.

image.png (1,907×396 px, 90 KB)

Event Timeline

dcaro created this task.Jul 6 2026, 8:15 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 6 2026, 8:15 AM
dcaro triaged this task as Medium priority.Thu, Jul 16, 12:15 PM
dcaro renamed this task from [toolsbeta,infra,logging] We are reaching 86% of quota by objects to [toolsbeta,infra,logging] We are reaching 96% of quota by objects.Thu, Jul 16, 12:48 PM
dcaro moved this task from Backlog to Todos on the tools-platform-team board.
dcaro updated the task description. (Show Details)
dcaro assigned this task to Raymond_Ndibe.Thu, Jul 16, 1:55 PM
dcaro moved this task from Todos to In progress on the tools-platform-team board.
Stashbot added a comment.Thu, Jul 16, 1:55 PM

Mentioned in SAL (#wikimedia-cloud) [2026-07-16T13:55:57Z] <Raymond_Ndibe> increased the object count quota to 500k T431248

Raymond_Ndibe added a comment.Thu, Jul 16, 2:02 PM

So we investigated this, and as far as we know, this issue seems to be coming from the tool-sample-complex-app tool in toolsbeta, because of the cronjobs that run every minute or so (cronjob, generate-emails).

We can decide if we want to reduce the frequency of these cronjobs, but for now, increasing the buckets object count quota to 500k works, so that was what we did.

Raymond_Ndibe closed this task as Resolved.Thu, Jul 16, 2:03 PM
Raymond_Ndibe moved this task from In progress to Done on the tools-platform-team board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits