Today einsteinum was going to run out of disk space, turns out this is due to puppet spam (!) in daemon.log. There's icinga config files diffs emitted at each puppet run causing churn, e.g:
```lang=diff, lines=5
define service {
-# --PUPPET_NAME-- cp1059 varnish http maps-frontend - port 3123
+# --PUPPET_NAME-- elastic2010 puppet_checkpuppetrun
active_checks_enabled 1
- check_command check_http_varnish!varnishcheck!3123
+ check_command nrpe_check!check_puppet_checkpuppetrun!10
check_freshness 0
check_interval 1
check_period 24x7
contact_groups admins
- host_name cp1059
+ host_name elastic2010
is_volatile 0
max_check_attempts 3
notification_interval 0
@@ -386662,19 +386662,19 @@
notification_period 24x7
passive_checks_enabled 1
retry_interval 1
- service_description Varnish HTTP maps-frontend - port 3123
- servicegroups cache_maps_eqiad
+ service_description puppet last run
+ servicegroups elasticsearch_codfw
}
```
I've archived the previous big daemon.log in `/srv` so there's no immediate danger for the next few days.
Aside from this issue there seem to be organically big log files, `syslog` `messages` and `user.log` have mostly the same contents (i.e. logs from icinga)
```
root@einsteinium:/var/log# du -hcs * | grep G
9.2G account
2.0G daemon.log.1
2.1G icinga
1.4G messages
2.7G messages.1
2.0G syslog
3.2G syslog.1
1.4G user.log
2.2G user.log.1
29G total
```