Page MenuHomePhabricator

Send some LibreNMS alerts to dcops and netops only
Closed, ResolvedPublic0 Estimated Story Points

Description

Some of the LibreNMS alerts should ideally open tasks automatically, 2nd best option for them is to email dcops and me. Instead of the current "email noc@".

The one I have in mind are:

  • Juniper environment status
  • interface errors
  • Port with no description on access switch
  • Port down for 7 days
  • Duplicate IP on mgmt network

See runbooks on https://wikitech.wikimedia.org/wiki/Network_monitoring#LibreNMS_alerts

If DCops agrees, the first step would be to create a dcops@wikimedia.org alias.

Related Objects

Event Timeline

ayounsi created this task.

I prefer we open tasks for anything requiring actual work, since yet another inbox spam is just that, more spam.

Also there is already a google group for 'Data Center Ops' datacenterops@wikimedia.org so you dont really have to create anything. (I suppose an email is required for this? Seems silly to email at all of we are generating tasks.)

I'd also prefer it, but implementing it is not a trivial task.
Emails to DCops would still be a step forward compared to the current situation where most of SRE is emailed (noc@) and I'm the one opening the tasks :)

so I'd just email the google group. Then the default settings for the folks in that (DC ops) is to get email updates (unless they have disabled it.)

ayounsi claimed this task.

I made the change to email dcops + me for those alerts.

All have a linked runbook.

The alerting might need to be fine tuned, but most if not all the latest alerts have been actionable. Please reopen if any issues.