Average check latency on icinga1001  appears to swing between 6 and 15 seconds. Latency larger than 10 seconds is not ideal .
Daniel and I have implemented some tweaks to help alleviate the problem, but cannot definitively answer whether or not it was successful. I think it might be worthwhile to gather these metrics into grafana as all the evidence so far is fairly anecdotal.
The "max_concurrent_checks" setting is a major factor for performance and is now configurable in Hiera and set to "0" (unlimited) on the new server while it is set to 10000 on the old server. .
Reference is made to section 7 in the tuning guide . which says "If you are seeing high latency values (> 10 or 15 seconds) for the majority of your service checks (via the extinfo CGI), you are probably starving Nagios of the checks it needs."