Nice :)

Trying to get the alerts to the prod alertmanagers I think is more convenient to me, mainly so the alerts with pages can be handled all from the same place.

Now having trusted prometheus seems the easiest, as that way we don't need to implement anything out of the standard, but pulling the alerts to prod instead seems a bit safer (so cloud still does not have a way to communicate into prod).

This might be interesting to get the interesting series from metricsinfra to prod (https://prometheus.io/docs/prometheus/latest/federation/)

Change 829746 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] p::metricsinfra:haproxy: Allow exposing federation endpoints

https://gerrit.wikimedia.org/r/829746

Change 829756 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] p::wmcs:prometheus: Add cloudvps federation job

https://gerrit.wikimedia.org/r/829756

@fgiunchedi This will work for the time being, though we will want to be able to allow volunteers (ex. toolforge roots) to silence some of these alerts, for example, when doing some maintenance on toolforge.

Some things that come to mind might be:

• Creating an "alert puller" to import alerts directly from the metricsinfra alertmanager, that way any alert silenced on metricsinfra (the VM) could be silenced also on the prod alertmanager.
• Creating a service that allows silencing the "cloud" alerts from within cloud realm.
• Move the alerting from within metricsinfra, adding the api keys or similar in there (having a fully duplicated stack).

WDYT? I can open a task to discuss this further (volunteers being able to silence alerts for volunteer managed projects).

In T313031#8320378, @dcaro wrote:

@fgiunchedi This will work for the time being, though we will want to be able to allow volunteers (ex. toolforge roots) to silence some of these alerts, for example, when doing some maintenance on toolforge.

Some things that come to mind might be:

• Creating an "alert puller" to import alerts directly from the metricsinfra alertmanager, that way any alert silenced on metricsinfra (the VM) could be silenced also on the prod alertmanager.
• Creating a service that allows silencing the "cloud" alerts from within cloud realm.
• Move the alerting from within metricsinfra, adding the api keys or similar in there (having a fully duplicated stack).

WDYT? I can open a task to discuss this further (volunteers being able to silence alerts for volunteer managed projects).

Yes thank you, let's followup/discuss in a separate task!

Created T320973 to follow up :)

Change 829746 abandoned by David Caro:

[operations/puppet@production] p::metricsinfra:haproxy: Allow exposing federation endpoints

Reason:

We found a nicer way of doing this :) (see the task)

https://gerrit.wikimedia.org/r/829746

Change 829756 abandoned by David Caro:

[operations/puppet@production] p::wmcs:prometheus: Add cloudvps federation job

Reason:

We found a nicer way of doing this :)

https://gerrit.wikimedia.org/r/829756