The last time we had to reinstall neon (icinga server) back in late July 2014, it became clear that it wasn't fully puppetized. The data on the post-puppet fixups and such there was only recorded in an email thread, which I'm pasting here now for posterity:
Pre-existing Puppetization Issues: ============================ This docs some stuff I had to figure out and apply manually on the host to get puppet happy and various icinga bits functional again. Most of it probably needs puppetization: a2enmod authnz_ldap a2enmod rewrite rm -f /var/lib/nagios/rw/nagios.cmd # root@neon:/var# find . -user nagios # ./cache/icinga/objects.cache # ./log/icinga # ./log/icinga/icinga.log # ./log/icinga/archives # ./lib/nagios cd /var; find . -user nagios|xargs chown -R icinga # From modules/ishmael/manifests/init.pp comments: cd /srv ; git clone https://github.com/asher/ishmael.git cd ishmael/; git clone https://github.com/asher/ishmael.git sample # For some nrpe checks to run: apt-get install libssl0.9.8 # To get irc bot to read logs: chmod o+r /var/log/icinga This one I fixed in puppet already since it was relatively simple (tcpircbot group): https://gerrit.wikimedia.org/r/#/c/150598/
Also, some new unpuppetization was added in the process:
1) I moved some of the fast, tiny write traffic from icinga onto a small 128MB tmpfs filesystem. This offloads a bunch of disk i/o. The gerrit changes to icinga.cfg are here: https://gerrit.wikimedia.org/r/150695 https://gerrit.wikimedia.org/r/150702 The mountpoint /var/icinga-tmpfs and the fstab entry for it aren't puppetized yet. The fstab entry is currently: tmpfs /var/icinga-tmpfs tmpfs size=128m,uid=icinga,gid=icinga,mode=755 0 0 (In retrospect, that may be a pretty silly mountpoint. Maybe put it elsewhere while puppetizing?) 2) I manually raised the default 1GB swap parition from the new install's setup to 8GB (via lvm) because we were running out of swap and oomkilling.