Page MenuHomePhabricator

Request to increase object storage capacity - Wikilink project
Closed, ResolvedPublic

Description

Project Name: Wikilink
Type of quota increase requested: Object Storage capacity
Amount to increase: Capacity to store ~20K files that on average weigh 700 KBs in two separate containers

  • Back of the napkin calculations: Every day, we generate ~3 objects that weigh about 700 KB each. We have a backlog of 5.5 years of historic data, which accounts for about 2200 objects that we haven't uploaded to Object Storage yet. If we want to store 10 years' worth of data, we would need 3objects*365days*10years = 10,950 objects, each weighing 700 KB.
  • For the aggregates data, we still haven't done archiving and file compression, but I would assume it would be slightly less storage needed 2objects*365days*10years = 7,300 objects

Reason: We are archiving link events that have been processed for the Wikilink platform. We want to keep the historic data in the Object storage so we don't overwhelm our small MariaDB Docker instance. We are also planning on archiving the historic aggregated data of the former for the same reasons.

Event Timeline

Peachey88 renamed this task from Request to increase object storage capacity to Request to increase object storage capacity - Wikilink project.Apr 29 2025, 8:51 AM
Peachey88 added a project: Wikilink-Tool.

Could you please update the request to have some concrete numbers - i.e. how much space and how many objects in total do you see you'll need, taking future growth into account as well?

I have updated the task description for the approximate amount we would need.

Update: We uploaded most of our historical data, and it seems we have run out of storage. We have hit the file limit, not the space limit.

Thanks for the numbers, they are very useful for us to be able to plan and manage the service, LGTM +1

Just to be clear, the requested quota bump is for:

  • max number of objects: 20K
  • max size: 20K * 700Kb = 14000000 Kb = 14 GB

Mentioned in SAL (#wikimedia-cloud) [2025-05-09T09:06:11Z] <arturo> bump object storage quota, with: aborrero@cloudcontrol1006:~$ sudo radosgw-admin quota set --quota-scope=user --uid wikilink\$wikilink --max-size=15GB --max-objects=20000 (T392870)

aborrero claimed this task.

I set 15GB instead of 14GB on purpose, so they have some extra room.

Many thanks! This should be more than enough for the long term!