In the puppet-private repo:
commit f94d1d4501e54136766722074b6821502213ccc1 staged on T367370_prom_alerts branch for pfw config.
commit 064db8663032904ceeaa6b1a32fc6ab93343dcc4 staged on T367370_prom_alerts branch for iptables config.

Something else I wanted to add: with respect to authoring and deploying alerts we have essentially centralized all alerts in operations/alerts.git repository. Said repo already contains scaffolding such as CI/tests integration and if you'd like to also commit FR alerts in there that's no problem; deployment is straightforward in the sense that in production we clone the repo and then selectively deploy alerts based on user-provided directions (#deploy comments at the top of the file). HTH

Change #1202827 had a related patch set uploaded (by Jgreen; author: Jgreen):

[operations/puppet@production] nsca_frack.cfg.erb remove deprecated check_endpoints service check

https://gerrit.wikimedia.org/r/1202827

@fgiunchedi We (fr-tech) are getting close to live testing with some alerts. We have started to build a set of alerts and are firing them off to our local alertmanager instance that will just send us email. With a config change, we could start pointing those at the production alertmanager instance.

We think the next logical step would be to set up the contact groups (team?) within alertmanager so that we can get them routed correctly via email/irc/splunk on-call. We want to make sure that we tag our alerts properly so that we don't cause issues for other folks. Is this something you can assist us with/point us in the right direction for?

Additionally, we have one alert that we saw today that is routed only through email (https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/refs/heads/master/team-netops/fundraising.yaml) that we'd like to get routed via splunk on-call to page us also. Perhaps that will be possible after the contact groups are set up.

Thank you for following up @Dwisehaupt and I'm glad to know you are making progress! These days I am no longer part of Observability, and will defer to @hnowlan to take it from here

Hi @Dwisehaupt,

In T367370#11361652, @Dwisehaupt wrote:

@fgiunchedi We (fr-tech) are getting close to live testing with some alerts. We have started to build a set of alerts and are firing them off to our local alertmanager instance that will just send us email. With a config change, we could start pointing those at the production alertmanager instance.

We think the next logical step would be to set up the contact groups (team?) within alertmanager so that we can get them routed correctly via email/irc/splunk on-call. We want to make sure that we tag our alerts properly so that we don't cause issues for other folks. Is this something you can assist us with/point us in the right direction for?

Regarding the contact groups, this depends on the Alertmanager configuration and there's two outcomes:

1. For a team defined in the config: In this case we need to implement new receivers which define the alert destinations based on specific thresholds (for ex. An IRC/Slack channel).
2. For a new team: In this case we need to create a new matching rule in the Alertmanager configuration along with the desired receivers for it.

In order to not cause alert issues for other teams the correct team and severity must be used as this is used by Alertmanager to properly route alerts to the desired destinations. I'm happy to help either by reviewing your patches on this or sending them myself if you'd like assistance with this. Additionally, here's our Alertmanager user guide.

Additionally, we have one alert that we saw today that is routed only through email (https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/refs/heads/master/team-netops/fundraising.yaml) that we'd like to get routed via splunk on-call to page us also. Perhaps that will be possible after the contact groups are set up.

That alert is handled by the following match on the Alertmanager configuration. Based on the match and severity that alert would be sent to the following receivers:

• sre-irc
• infrastructure-foundations-irc

I don't see any email receivers matching for that alert however, we can set-up other receivers that would (for example) send that alert to a Slack channel.

Regarding the page for that alert, there are two outcomes:

1. The alert must page SRE: In this case there's already a receiver for SRE Pages. In order for it page all that's needed is to change the severity from critical to page.
2. The alert must page another team: In this case we need to implement a routing key on SplunkOnCall, then add a new receiver for that routing key on the alertmanager config.

I can help you to create the routing key on SplunkOnCall if you give me with the team name.

Change #1204648 had a related patch set uploaded (by Dwisehaupt; author: Dwisehaupt):

[operations/puppet@production] Alertmanager: Add fr-tech-ops and update fr-tech groups

https://gerrit.wikimedia.org/r/1204648

@andrea.denisse Thanks for the feedback. We want to mimic what we currently have for the icinga alerting groups. I think I have done that with this commit (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1204648). It goes a little further to split out fr-tech and fr-tech-ops since we have some hardware/OS level alerts that the whole fr-tech team doesn't need to have.

Our alert rules will live in fundraising and fire from our prometheus server. Given that, I believe we can reference the team in our rules to make sure that they get properly routed. Here is an example of one our our rules we have been testing locally:

---

groups:

  - name: queue-alerts

    rules:

      - &redis_queue_size_warn
        alert: RedisQueueSize
        expr:
          redis_queue_total{
              cluster="frqueue",
              queue=~"(contribution_tracking|payments_init|pending|refund)"}
            > 1500
        for: 5m
        labels:
          severity: warning
          team: 'fr-tech'
        annotations:
          description: 'High redis queue size'
          summary: "Redis Queue {{ $labels.queue }} is high: [{{ $value }}]"
          dashboard: 'https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc'
      - &redis_queue_size_crit
        alert: RedisQueueSize
        expr:
          redis_queue_total{
              cluster="frqueue",
              queue=~"(contribution_tracking|payments_init|pending|refund)"}
            > 2000
        for: 5m
        labels:
          severity: critical
          team: 'fr-tech'
        annotations:
          description: 'Critical redis queue size'
          summary: "Redis Queue {{ $labels.queue }} is Critical: [{{ $value }}]"
          dashboard: 'https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc'

Does this make sense?

As far as the PfwCoreBGPDown alert, when reading through the config I think we are getting email alerts due to this rule. When we have the new groups set up, maybe we can have that route to our paging level.

In T367370#11368418, @Dwisehaupt wrote:

@andrea.denisse Thanks for the feedback. We want to mimic what we currently have for the icinga alerting groups. I think I have done that with this commit (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1204648). It goes a little further to split out fr-tech and fr-tech-ops since we have some hardware/OS level alerts that the whole fr-tech team doesn't need to have.

Our alert rules will live in fundraising and fire from our prometheus server. Given that, I believe we can reference the team in our rules to make sure that they get properly routed. Here is an example of one our our rules we have been testing locally:

---

groups:

  - name: queue-alerts

    rules:

      - &redis_queue_size_warn
        alert: RedisQueueSize
        expr:
          redis_queue_total{
              cluster="frqueue",
              queue=~"(contribution_tracking|payments_init|pending|refund)"}
            > 1500
        for: 5m
        labels:
          severity: warning
          team: 'fr-tech'
        annotations:
          description: 'High redis queue size'
          summary: "Redis Queue {{ $labels.queue }} is high: [{{ $value }}]"
          dashboard: 'https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc'
      - &redis_queue_size_crit
        alert: RedisQueueSize
        expr:
          redis_queue_total{
              cluster="frqueue",
              queue=~"(contribution_tracking|payments_init|pending|refund)"}
            > 2000
        for: 5m
        labels:
          severity: critical
          team: 'fr-tech'
        annotations:
          description: 'Critical redis queue size'
          summary: "Redis Queue {{ $labels.queue }} is Critical: [{{ $value }}]"
          dashboard: 'https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc'

Does this make sense?

As far as the PfwCoreBGPDown alert, when reading through the config I think we are getting email alerts due to this rule. When we have the new groups set up, maybe we can have that route to our paging level.

Hi @Dwisehaupt , the routes and the patch LGTM.

I also checked SplunkOnCall and there's already a Fundraising team there with you and @Jgreen so the pages should work.

The Puppet repo has a special set of permissions so please let me know if you need help merging when ready.

Hi @Dwisehaupt , the routes and the patch LGTM.

I also checked SplunkOnCall and there's already a Fundraising team there with you and @Jgreen so the pages should work.

The Puppet repo has a special set of permissions so please let me know if you need help merging when ready.

@andrea.denisse Thanks for the review! We (fr-tech-ops) don't have rights to deploy changes to the prod puppet repo. Feel free to merge and deploy the change when it's convenient for you.

Change #1204648 merged by Andrea Denisse:

[operations/puppet@production] Alertmanager: Add fr-tech-ops and update fr-tech groups

https://gerrit.wikimedia.org/r/1204648

Change #1202827 merged by Herron:

[operations/puppet@production] nsca_frack.cfg.erb deprecate check_endpoints service and pay-lvs hostgroup

https://gerrit.wikimedia.org/r/1202827

We are looking to start firing alerts through alertmanager after the Q3 maintenance window (2026-02-02 - 2026-0206) is complete. We should be able to transition fairly quickly after we verify the alerts route correctly.

@andrea.denisse Just wanted to follow up and let you know that we are close to ready for the transition. We have all of the necessary checks running against our alertmanager system and will be prepping the config on our end to point the alerts at the prod alertmanager instance shortly.

Is there a time that works for you that we could start pointing the alerts at the prod instance? I figure that we will want to trigger some alerts to make sure that they fire through properly.

Adjusted the prometheus config in frack to sent alerts to the production alertmanager. Alerts are properly showing up in victorops and via email. Not seeing alerts in #wikimedia-fundraising irc channel at this point.

Done today:

• alerts pointed at production alertmanager
• existing nagios checks now sending dummy check responses to keep icinga all green

Things to do/adjust to:

• remove icinga host/service entries from prod icinga config
• disable the nagios cronjob on all hosts
• remove nagios config and packages from all hosts
• determine pathway to managing alerts (acknowlegement/silence) given both alertmanager and victorops are in play now.
• train up fr-tech on how to manage alerts in the new framework

Last portion is onwiki documentation on acknowledging alerts.

I walked fr-tech through the process in tech talk on May 20th - https://drive.google.com/file/d/1lQifnyicKqpZ3GHNRdheEjC7Mk7V_4RU/view

There is a basic knowledge of how it is done, but documentation would be extra nice.