Page MenuHomePhabricator

[metricsinfra.configurator] Allow adding extra labels to alerts, to be able to filter the ones with 'team="wmcs"'
Closed, ResolvedPublic

Description

The current need is that now that we have metricsinfra alerts showing up on prod alertmanager (alerts.wikimedia.org), we (wmcs, the ones responsible for monitoring those alerts) want to be able to filter them easily by adding the label 'team="wmcs"'.

That will enable also paging through victorops for critical alerts.

Given that, the idea is to enable adding custom labels at the project level to all the projects.

The restrictions are:

  • Not every project should be able to add the 'team=wmcs' alert, as not every project should be monitored/page us

Restrict the labels that can be added by user role

This means having a list of labels that are restricted, and limit to global admins the right to add this labels. Allowing any other admins to add their own custom labels.

Allow only adding the 'team' label, and restrict who can add 'wmcs'

Similar as the above, but only allowing to add the 'team' label.

Allow only a subset of labels, and restrict who can add 'team=wmcs'

Get a list of labels that you can add at the project label, ex:

  • team
  • service
  • any others?

And restrict who can add 'team=wmcs' as above.

Allow adding labels only to cloud roots

Do not allow anyone to add any extra labels, except to the cloud roots.

Current task

Given that we don't have yet an interface for users to interact with it (except roots going to the DB), this task is to add support for specifying any labels at the project level, and will be decided and implemented later when we have a user interface, if we only allow only cloud roots or any user, or any more fine grained control.

Related Objects

Event Timeline

Change 909682 had a related patch set uploaded (by David Caro; author: David Caro):

[cloud/metricsinfra/prometheus-manager@master] lavender: run plain with default options

https://gerrit.wikimedia.org/r/909682

Change 909683 had a related patch set uploaded (by David Caro; author: David Caro):

[cloud/metricsinfra/prometheus-manager@master] build: Add missing __init__.py so it can find the migrations

https://gerrit.wikimedia.org/r/909683

Change 909684 had a related patch set uploaded (by David Caro; author: David Caro):

[cloud/metricsinfra/prometheus-manager@master] project: add extra_labels column

https://gerrit.wikimedia.org/r/909684

Change 909685 had a related patch set uploaded (by David Caro; author: David Caro):

[cloud/metricsinfra/prometheus-manager@master] dev: add some notes on how to setup the dev environment

https://gerrit.wikimedia.org/r/909685

Change 909682 merged by jenkins-bot:

[cloud/metricsinfra/prometheus-manager@master] lavender: run plain with default options

https://gerrit.wikimedia.org/r/909682

Change 909683 merged by jenkins-bot:

[cloud/metricsinfra/prometheus-manager@master] build: Add missing __init__.py so it can find the migrations

https://gerrit.wikimedia.org/r/909683

Change 909684 merged by jenkins-bot:

[cloud/metricsinfra/prometheus-manager@master] project: add extra_labels column

https://gerrit.wikimedia.org/r/909684

Change 909685 merged by jenkins-bot:

[cloud/metricsinfra/prometheus-manager@master] dev: add some notes on how to setup the dev environment

https://gerrit.wikimedia.org/r/909685

Change 911893 had a related patch set uploaded (by David Caro; author: David Caro):

[cloud/metricsinfra/prometheus-configurator@master] prometheus: add project level extra_labels support

https://gerrit.wikimedia.org/r/911893

Change 912287 had a related patch set uploaded (by David Caro; author: David Caro):

[cloud/metricsinfra/prometheus-manager@master] api: show extra_labels in basic project info

https://gerrit.wikimedia.org/r/912287

Change 912287 merged by jenkins-bot:

[cloud/metricsinfra/prometheus-manager@master] api: show extra_labels in basic project info

https://gerrit.wikimedia.org/r/912287

Change 911893 merged by jenkins-bot:

[cloud/metricsinfra/prometheus-configurator@master] prometheus: add project level extra_labels support

https://gerrit.wikimedia.org/r/911893

Change 913117 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::prometheus: set team: wmcs on alerts

https://gerrit.wikimedia.org/r/913117

Change 913117 merged by David Caro:

[operations/puppet@production] P:toolforge::prometheus: set team: wmcs on alerts

https://gerrit.wikimedia.org/r/913117