Hi,

Looking at our Alertmanager configuration, we currently send Slack notifications by creating an email address for a channel and sending alerts to it via email. However, Alertmanager can also send alerts directly to Slack using a webhook.

@colewhite, are there any downsides to using the webhook method instead of email?

@Mooeypoo , do you know if these alerts are already present in Prometheus?

Hi @hnowlan, I noticed the parent task is T401908. Is the goal here to ingest Grafana alerts into Alertmanager before sending them to Slack, or to route the alerts Alertmanager already receives into Slack channels?

In T401730#11088178, @andrea.denisse wrote:

@colewhite, are there any downsides to using the webhook method instead of email?

I believe this isn't possible because the grafana hosts cannot connect outside the production network. They can send emails through our internal mail servers, though.

In T401730#11090237, @colewhite wrote:

In T401730#11088178, @andrea.denisse wrote:

@colewhite, are there any downsides to using the webhook method instead of email?

I believe this isn't possible because the grafana hosts cannot connect outside the production network. They can send emails through our internal mail servers, though.

But the Alertmanager hosts can connect outside of production so I think that the webhook can be used. The Alertmanager hosts already communicate with SplunkOnCall along with the Prometheus dead man switch.

As for the Grafana alerts, I think that they can be routed to Alertamanger and then Alertmanager could send the alerts to Slack using the webhook.

In T401730#11088178, @andrea.denisse wrote:

@Mooeypoo , do you know if these alerts are already present in Prometheus?

We don't yet have any alerts set up, but if we need to we can definitely come up with one we'd need and can test with,

In T401730#11090663, @andrea.denisse wrote:

In T401730#11090237, @colewhite wrote:

In T401730#11088178, @andrea.denisse wrote:

@colewhite, are there any downsides to using the webhook method instead of email?

I believe this isn't possible because the grafana hosts cannot connect outside the production network. They can send emails through our internal mail servers, though.

But the Alertmanager hosts can connect outside of production so I think that the webhook can be used. The Alertmanager hosts already communicate with SplunkOnCall along with the Prometheus dead man switch.

As for the Grafana alerts, I think that they can be routed to Alertamanger and then Alertmanager could send the alerts to Slack using the webhook.

Hi, I created a Slack webhook to test this and can confirm that the alert hosts are able to communicate with Slack via the webhook, so it’s feasible to use it.

Implementing it would mainly depend on whether the team considers this a suitable option and on obtaining IT approval to register the application in our Slack workspace. If we decide to move forward, we could even migrate the current channels where Alertmanager sends notifications via email to instead use the webhook.

In T401730#11091376, @Mooeypoo wrote:

In T401730#11088178, @andrea.denisse wrote:

@Mooeypoo , do you know if these alerts are already present in Prometheus?

We don't yet have any alerts set up, but if we need to we can definitely come up with one we'd need and can test with,

That's okay. Just to clarify, these alerts are expected to be Prometheus alerts, correct?

Another question, do you know the name of the channel that would receive these notifications?

We just opened #api-alerts on slack, so we can use that channel.

And yes, we're talking about Prometheus alerts. Thank you!

Hi folks,

Updating on the task status.
I was able to send alerts to the #api-alerts channel using a Slack webhook so we can use that method however, the drawback with using a webhook is that they're channel bounded so we would need to create a webhook every time we want to implement this functionality for other channels.

To overcome the limitation of webhooks I'm testing the authentication process using a bot token as it seems that using one could allow us to send the messages to any channel defined in the Alertmanager route without having to define a webhook per-channel. This would greatly simplify both secret management along with the set-up of new alert routes.

I sent a request to the IT team for the Alertmanager app to authenticate using OAuth Tokens, once it's granted I'll continue testing and post the outcome here.

Mentioned in SAL (#wikimedia-operations) [2025-09-03T23:38:37Z] <denisse> Adding slack_bot_token to private repo - T401730

Change #1184611 had a related patch set uploaded (by Andrea Denisse; author: Andrea Denisse):

[operations/puppet@production] alert: Add Slack route to send Prometheus alerts

https://gerrit.wikimedia.org/r/1184611

Hi folks, I’ve updated the Wikitech documentation for this feature. I’d really appreciate your feedback: https://wikitech.wikimedia.org/wiki/Alertmanager#Sending_alerts_to_Slack

Change #1184611 merged by Andrea Denisse:

[operations/puppet@production] alert: Add Slack route to send Prometheus alerts

https://gerrit.wikimedia.org/r/1184611