noticed many of these for unrelated reasons in neon:/var/log/icinga/icinga.log
[1415692015] Max concurrent service checks (3200) has been reached. Nudging mw1178:nutcracker process by 11 seconds...
root@neon:/var/log/icinga# grep -ic 'max concurrent' icinga.log* icinga.log:25337 icinga.log.1:19954 icinga.log.2:12637 icinga.log.3:6705 icinga.log.4:25447 icinga.log.5:15188 icinga.log.6:33015 icinga.log.7:7193
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | akosiaris | T1242 icinga "max concurrent checks" limits reached | |||
| Resolved | akosiaris | T85222 improve icinga performance / solve general load issues on neon |
It is possible to increase the limit but it does increase CPU and according to ganglia, neon's already averaging about 93%.
If neon == https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000223.eqiad.wmflabs I would just assign it some more cpu's and memory to give it more breathing space, this vm is tiny.
This has been resolved. The limit has been raised since the deprecation of check_sslXNN and the migration of neon to einsteinium/tegmen which provided us with more resources.