Page MenuHomePhabricator

Make contact group for Netbox report alerts
Open, Needs TriagePublic

Description

Basically we need the alerts from netbox reports to go to the DCops IRC channel instead of -operations, so a new contact group and bot are (perhaps) needed unless an existing one can be used.

Event Timeline

crusnov created this task.Mon, Aug 19, 3:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Aug 19, 3:01 PM
Dzahn added a subscriber: Dzahn.Wed, Sep 11, 10:18 PM

The existing bot, icinga-wm, can be used.

What is needed is:

  • create 2 custom notification commands (modules/nagios_common/templates/notification_commands.cfg.erb), notify-service-by-irc-dcops and notify-host-by-irc-dcops, so one for services and one for hosts. copy the commandline from existing "by-irc" commands but make sure the output gets appended to a new log file, /irc-dcops.log
  • create a new Icinga contact (private repo, modules/secret/secrets/nagios/contacts.cfg), irc-dcops. copy an existing "irc-" contact but adjust host_notification_command service_notification_command to use your new commands (and logfile)
  • create a new Icinga contactgroup (public repo, modules/nagios_common/files/contactgroups.cfg) for datacenter ops and add the special contact you created to it (and optional the human members of dcops so they get notified too), check icinga config is ok after running puppet (icinga -v /etc/icinga/icinga.cfg)
  • in puppet identify the monitoring::service / nrpe::monitor_service classes that are relevant and add the new contactgroup as a parameter to them
  • on the Icinga server go to /var/log/icinga/ and check if the new logfile has been created (you may have to touch it manually the very first time) and alerts get logged to it
  • configure ircecho (modules/profile/manifests/icinga/ircbot.pp) to map the logfile to your IRC channel ($ircecho_logs, see existing examples) (restart ircecho?)

Unless i forgot something now icinga-wm should join the channel once the first new event is logged to file. You can also test it by just echo'ing something into the logfile manually to make the bot say stuff.

Added this on Wikitech because it had no docs. https://wikitech.wikimedia.org/wiki/Icinga#IRC_bot