/var/log/account/pacct is 13GB on the k3s instance, meanwhile the root partition is 20GB.
Unsure when/why/how process accounting started (and/or why there's no logrotate for it).
Either way: we should decide if we want to keep it and if so logrotate, if not delete and remove pacct service. It's likely safe to get rid of and unless there's a compelling reason to keep the default should be remove.
High priority since if we don't do it then it seems likely we'll run out of disk space and PatchDemo/Catalyst will all grind to a halt.
Added logrotate configuration at /etc/logrotate.d/acct:
/var/log/account/* {
weekly
rotate 8
copytruncate
compress
notifempty
missingok
dateext
}Logs are being rotated and root partition usage has plummeted:
$ df -h | grep sda1 /dev/sda1 20G 4.0G 15G 22% /
Some useful notes.
It looks like this is an known issue for WMCS hosts:
Something interesting: That task above created some Alertmanager alerts https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/25/diffs. Recently we've started talking about adding monitoring/alerting to the Catalyst cluster, it seems Alertmanager would be the way to integrate with the foundation's Prometheus infra