The python stats collector diamond does not compress its log under /var/log/diamond/ . That might lead to /var/ filling up entirely (it is only 2GB).
We might need diamond to compress logs, or a cronjob to gzip them if it can't compress them. Also want to make sure we dont keep too many archives.
Version: unspecified
Severity: normal
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T71590 rsync errors to beta cluster, inconsistent state after scap | |||
| Resolved | hashar | T71601 Log files on labs instance fill up disk (/var is only 2GB) (tracking) | |||
| Declined | None | T71602 diamond does not compress its logs |
This used to happen really consistently (see bug 66458). We 'fixed' it by essentially making diamond log only on errors, which shouldn't happen often. The role also realm-branches in labs to only keep one day of logs, vs prod where we keep more. Closing as wontfix for now.
You might have leftover archive.log and diamond.log files from before the fixes. If so, feel free to rm -rf them, diamond won't re-create.