Page MenuHomePhabricator

icinga "max concurrent checks" limits reached
Closed, ResolvedPublic

Description

noticed many of these for unrelated reasons in neon:/var/log/icinga/icinga.log

[1415692015] Max concurrent service checks (3200) has been reached.  Nudging mw1178:nutcracker process by 11 seconds...
root@neon:/var/log/icinga# grep -ic 'max concurrent' icinga.log*
icinga.log:25337
icinga.log.1:19954
icinga.log.2:12637
icinga.log.3:6705
icinga.log.4:25447
icinga.log.5:15188
icinga.log.6:33015
icinga.log.7:7193

Event Timeline

fgiunchedi raised the priority of this task from to Needs Triage.
fgiunchedi updated the task description. (Show Details)
fgiunchedi added a project: acl*sre-team.
fgiunchedi changed Security from none to None.
fgiunchedi added a subscriber: fgiunchedi.
fgiunchedi triaged this task as Medium priority.Dec 19 2014, 2:28 PM
fgiunchedi added a project: observability.

It is possible to increase the limit but it does increase CPU and according to ganglia, neon's already averaging about 93%.

If neon == https://wikitech.wikimedia.org/wiki/Nova_Resource:I-00000223.eqiad.wmflabs I would just assign it some more cpu's and memory to give it more breathing space, this vm is tiny.

akosiaris claimed this task.
akosiaris added a subscriber: akosiaris.

This has been resolved. The limit has been raised since the deprecation of check_sslXNN and the migration of neon to einsteinium/tegmen which provided us with more resources.