The cluebotng-staging tool is showing up as one of our largest users of disk space at this time:
559G /srv/tools/project/cluebotng-staging
Most of that seems to be log files. @DamianZaremba, @RichSmith, @Naomi, please fix your tool to generate or store much less output.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | taavi | T395000 2025-05-22 Toolforge NFS cleanup | |||
| Resolved | None | T395006 cluebotng-staging tool uses ~560G of disk space |
Post https://github.com/cluebotng/botng/commit/9683902f88341a0c4bf60ac3bbd6089d9724ae9f the logs are rotated daily.
They are quite large aa currently the code is running with trace level logging enabled.
It just needs a job adding to prune the older log files - I haven't had time to do this yet.
Feel free to remove any file botng-*.log from the home directory until a cleanup "cron" is added.
Alright, I've cleared all logs from March and April. Leaving this task open until you can implement some automatic cleanup, since this will come back to alert us otherwise. Thank you for the quick response!
Mentioned in SAL (#wikimedia-cloud) [2025-05-22T14:42:48Z] <taavi> cleanup 200G+ of old log files per T395006
https://github.com/cluebotng/botng/commit/89e2e6635beb88a7288f9f94e84335ac166f4395
Before:
tools.cluebotng-staging@tools-bastion-12:~$ du -shc botng-*.log 11G botng-20250501.log 11G botng-20250502.log 11G botng-20250503.log 11G botng-20250504.log 11G botng-20250505.log 12G botng-20250506.log 10G botng-20250507.log 11G botng-20250508.log 11G botng-20250509.log 11G botng-20250510.log 11G botng-20250511.log 11G botng-20250512.log 12G botng-20250513.log 11G botng-20250514.log 12G botng-20250515.log 11G botng-20250516.log 9.8G botng-20250517.log 11G botng-20250518.log 11G botng-20250519.log 11G botng-20250520.log 4.7G botng-20250521.log 215G total
Deployed:
tools.cluebotng-staging@tools-bastion-12:~$ kubectl get pods NAME READY STATUS RESTARTS AGE cbng-5699588d4-6ddvj 1/3 Terminating 400 34h cbng-6dfd8ff56b-ptfqh 3/3 Running 0 95s
After:
tools.cluebotng-staging@tools-bastion-12:~$ du -shc botng-*.log 11G botng-20250509.log 11G botng-20250510.log 11G botng-20250511.log 11G botng-20250512.log 12G botng-20250513.log 11G botng-20250514.log 12G botng-20250515.log 11G botng-20250516.log 9.8G botng-20250517.log 11G botng-20250518.log 11G botng-20250519.log 11G botng-20250520.log 4.7G botng-20250521.log 24M botng-20250522.log 131G total