Page MenuHomePhabricator

track growth of project and home directories
Closed, ResolvedPublic

Description

We have been filling storage on the tools NFS project share as of late, and it has grown 3% in less than 48 hours. We really need some historical perspective to track this over time.

Event Timeline

chasemp created this task.Feb 11 2016, 4:38 PM
chasemp raised the priority of this task from to Normal.
chasemp updated the task description. (Show Details)
chasemp added a project: Cloud-Services.
chasemp added subscribers: tom29739, valhallasw, Aklapper, chasemp.

Historical data is available in graphite.wm.o as servers.labstore1001.diskspace._srv_project_tools.byte_used:

The current growth rate is ~200GB/24h:

However, we probably want to know this on a user level. Could we enable quota on the NFS server to track per-user disk usage (but without actually enforcing quotas immediately)?

Change 335855 had a related patch set uploaded (by Madhuvishy):
labstore: Diamond collector to track tools home and project dir sizes

https://gerrit.wikimedia.org/r/335855

Change 335855 had a related patch set uploaded (by Madhuvishy):
labstore: Diamond collector to track directory sizes

https://gerrit.wikimedia.org/r/335855

Change 335855 merged by Madhuvishy:
labstore: Diamond collector to track directory sizes

https://gerrit.wikimedia.org/r/335855

Graphite now logs tools and misc NFS share directory sizes at labstore.labstore-secondary.DirectorySizeCollector on prod. Grafana dashboard based on this data - https://grafana.wikimedia.org/dashboard/db/labstore-nfs-directory-sizes

madhuvishy closed this task as Resolved.Feb 23 2017, 11:42 PM
madhuvishy claimed this task.

The dashboard is updating fine the last 3 days - closing this.