Page MenuHomePhabricator

Usability improvements for alerts.wikimedia.org (Karma)
Open, Needs TriagePublic

Description

The usability of alerts.wikimedia.org has long been a noted pain point for many SREs. The interface can be confusing, and there are many avenues for improvement.

This ticket is an overarching parent task for usability improvements we can make to the alerts interface.

Some basics that don't warrant subtasks:

  • Label ordering. Karma doesn't guarantee ordering, but we can use labels.order to ensure that we consistently order them (severity, team, instance being good first params)
  • Make links action buttons - makes runbooks and dashboards immediately accessible (already done)
  • Phab link support (This already exists in WMCS!)
  • Is there anything to be gained from changing our ack defaults? Longer silences? 15 minutes is not really useful for anyone, an hour is probably better.
  • Simplify the silence form - we currently have a lot of high-cardinality matchers in our silence form, and we can use silenceForm to drop some of these if they're not useful
  • Maybe a bit more contentious, but use labels.color.unique to colour by team? WMCS also already does this
  • Trim labels - there's a lot of clutter in each card, can we trim some info from labels without impairing our ability to silence usefully by label?
  • Are there improvements we could make to alert grouping? There are a bunch of config options for it.

Event Timeline

Reviewing our config, some things I think we could do easily that would make a difference

  • Label ordering. Karma doesn't guarantee ordering, but we can use labels.order to ensure that we consistently order them (severity, team, instance being good first params)
  • Make links action buttons - makes runbooks and dashboards immediately accessible
  • Phab link support (This already exists in WMCS!)
  • Is there anything to be gained from changing our ack defaults? Longer silences? 15 minutes is not really useful for anyone, an hour is probably better.
  • Simplify the silence form - we currently have a lot of high-cardinality matchers in our silence form, and we can use silenceForm to drop some of these if they're not useful
  • Maybe a bit more contentious, but use labels.color.unique to colour by team? WMCS also already does this
  • Trim labels - there's a lot of clutter in each card, can we trim some info from labels without impairing our ability to silence usefully by label?
  • Are there improvements we could make to alert grouping? There are a bunch of config options for it.

Some context and previous work towards this end in T284213

Change #1310141 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] karma: automatically linkify Phab task IDs in messages

https://gerrit.wikimedia.org/r/1310141

Change #1310519 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] karma: consistently order some labels first

https://gerrit.wikimedia.org/r/1310519

Change #1310141 merged by Hnowlan:

[operations/puppet@production] karma: automatically linkify Phab task IDs in messages

https://gerrit.wikimedia.org/r/1310141

Change #1311029 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] karma: strip some useless labels from panel display

https://gerrit.wikimedia.org/r/1311029