tracking ticket to improve icinga performance, for obvious reasons the biggest improvements can be obtained by tweaking the icinga active checks.
the machine hosting icinga (neon) routinely suffers from high load, occasionally icinga children won't cleanup their children leaving behind zombies and eventually swapping.
also given the rate at which we spawn checks fork() overhead is significant, thus for example checks that spawn other commands are incredibily inefficient (e.g. check_ssl_cert spawns perl at each invocation)
Neon is routinely at 0% idle CPU when looking at realtime info. Even after manually turning down some of the most CPU-expensive checks, the remaining load still routinely spiked the machine to 0% idle CPU, so in its normal config it's definitely well past the line.
Also, the main icinga process itself is single-threaded and routinely locks up a single CPU core, effectively running out of processing power to keep up with its own demands, even if other cores are idle.
I noticed this while debugging the intermittent ipv6 monitor failures. I really don't know if this is casual or even related, but I figure solving this basic issue seems prudent...