@bennofs The wdumps tool is currently the largest consumer of NFS storage in the tools project space. We're running short on available storage and wondering if it's possible to remove some of the recently created dumps.
In total the generated dumps are using 773 gigabytes of storage, any space you can help reclaim would be greatly appreciated.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • Bstorm | T261335 Fix alerting for disk space on the NFS servers | |||
| Resolved | Andrew | T261336 2020-08-26: tools NFS share cleanup | |||
| Resolved | • JHedden | T247315 2020-03-10: tools and misc nfs share cleanup | |||
| Duplicate | • Bstorm | T284964 2021-06-15: Tools NFS share cleanup | |||
| Resolved | bennofs | T247449 wdumps custom generated dumps storage space |
Toolforge has 8TB of storage for everyone including both users and tools. @bennofs's tool is currently using ~9% of all storage to host 123 data files. It is also not obvious if the tool has any cleanup mechanism at all for older files.
Thank you @bennofs. In the mean time can you please manually delete some of the custom dumps? We're at 90% storage utilization and fighting off our monitoring alarms.
wdumps is in the top 3 users of Toolforge storage again https://grafana.wikimedia.org/d/50z0i4XWz/tools-overall-nfs-storage-utilization?viewPanel=4&orgId=1
@bennofs were you able to automate the cleanup?
@bennofs, this tool is currently consuming about 225Gb of disk space. Can you please update us about your plan for automating cleanup? And, a quick by-hand cleanup would help keep things moving in the meantime.
Thanks!
@bennofs @mkroetzsch This needs to be cleaned up. You are now at 310.95 GB and tools NFS is filling up.
Please take manual action and do something to automate this as well. I can definitely go start deleting things, but I expect that will break the tool since I don't know what I'm doing.
Responses here seem to have been non-existent, so I reported in their github project as well: https://github.com/bennofs/wdumper/issues/31
I've moved the dump files to scratch for now, hope this helps. I also manually cleaned up some big dumps.
There is already a cron task that deletes dumps after 14 days. If this continues to be a problem, I could reduce the cleanup interval as well.
That puts scratch at the entirely-acceptable 63% and means the project share is now time :)
That was enough to clear the alerts, for sure.