In T419029#11675020, @EBernhardson wrote:

I was thinking of access as not solving the current issue, as we have a plan forward for that, but as more of addressing possibilities on a longer-term basis.  It seems like once or twice a year I run into something that would go easier if I had more access.  I see from the puppet data.yaml file that we have a couple, but very few, engineers with ops access. This isn't the first time the question of ops level access has come up, but in the past I've pushed off requesting access as it seemed not strictly necessary. It's still not strictly necessary, but I'm leaning towards this easing some of the work I do. The full solutions, like the readahead support being setup now, would still be the end-state we would be looking for, but the additional access would better allow figuring out where these things need to be before the full solution is ready to be deployed.

Historically we've used root access on the search fleet for a variety of reasons, sometimes debugging something amounts to attaching strace to a process, or using a kernel probe (these days it would probably be bpf) to print out arguments to a particular kernel function, utilizing the kernel page-types tool to evaluate file-cache effectiveness, or putting together a custom C program that reaches in with ptrace_do to execute syscalls inside the server process. These cases are rare, but I don't expect us to ever have general solutions to deep-diving into how something works exactly to understand why it's not working in the expected way. I've been doing something similar to the ptrace_do approach in recent days, but it requires using LD_PRELOAD to inject the custom code, which means i need to roll a cluster restart for each evaluation.

Essentially i noticed while trying to debug the opensearch servers in k8s recently that I don't have access to tools that I've used in the past, and it ends up being much more of a black box than it has to be.

In T390734#11687686, @Ben.buchenau wrote:

Hello guys - follow-up request regarding Kerebos authentication: Can I get a keytab file for my user?

I checked the Kerebos docs on Wikitech and saw this is the common way to go for automated services. I currently fetch aggregated data in a scheduled script using Spark for WMDE internal monitoring - but this is only semi-automatic since I have to kinit every second week. We are building an internal dashboard on top of that soon, Having a keytab file for fully automated authentification would be very practical for that.

Best, Ben (Data Analyst at WMDE)

Hi @EMcFarland-WMF , access to the analytics-privatedata-users is granted along with the kerberos principal. You should receive an email regarding the kerberos principal requesting you to change your password. Feel free to reopen if there's anything else I can help with.

In T413006#11471604, @Marostegui wrote:

@andrea.denisse I assume you'd handle this yourelf or you'd need help from clinic duty?

Maybe we should look into implementing a way for arclamp to create tasks when this issue happens.

In T411436#11426873, @andrea.denisse wrote:

In T411436#11425341, @SEgt-WMF wrote:

In case it is useful: the MediaWiki page @Rmaung pointed out was the first thing that came up for me when I googled "Request Access Superset Wikimedia", so it might be good to update that one too? In any case, I'm super happy to learn about the proper process, so thank you, @Aklapper!

Hi Silvia, could you please update the task with the fields from the https://phabricator.wikimedia.org/maniphest/task/edit/form/8/ form?

Hi folks, if I understand correctly the access request can't be fulfilled because the requested type of access doesn't exist.

Hi folks, the patch for this task is merged. Can we close it as resolved?

Closing as resolved, feel free to reopen if there's anything else I can assist with.

Closing as resolved, please let me know if there's anything else I can assist with.

Closing as resolved, please let me know if there's anything else I can assist with.

Closing as resolved, please let me know if there's anything else I can assist with.

I wrote to Zoe directly to confirm of this request.

Was collaboration-services tag removed? This tag showed up on the Clinic Duty dashboard.

In T411436#11425341, @SEgt-WMF wrote:

In case it is useful: the MediaWiki page @Rmaung pointed out was the first thing that came up for me when I googled "Request Access Superset Wikimedia", so it might be good to update that one too? In any case, I'm super happy to learn about the proper process, so thank you, @Aklapper!

Hi @KFrancis, I was unable to find @medelius on the NDA spreadsheet, could you please help me to confirm their NDA status?

Hi @VPuffetMichel , do you approve this request?

Hi Raine, this is on the clinic duty dashboard.

Hi Hugh, this is on the clinic duty dashboard.

In T367370#11368418, @Dwisehaupt wrote:

@andrea.denisse Thanks for the feedback. We want to mimic what we currently have for the icinga alerting groups. I think I have done that with this commit (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1204648). It goes a little further to split out fr-tech and fr-tech-ops since we have some hardware/OS level alerts that the whole fr-tech team doesn't need to have.

Our alert rules will live in fundraising and fire from our prometheus server. Given that, I believe we can reference the team in our rules to make sure that they get properly routed. Here is an example of one our our rules we have been testing locally:

---

groups:

  - name: queue-alerts

    rules:

      - &redis_queue_size_warn
        alert: RedisQueueSize
        expr:
          redis_queue_total{
              cluster="frqueue",
              queue=~"(contribution_tracking|payments_init|pending|refund)"}
            > 1500
        for: 5m
        labels:
          severity: warning
          team: 'fr-tech'
        annotations:
          description: 'High redis queue size'
          summary: "Redis Queue {{ $labels.queue }} is high: [{{ $value }}]"
          dashboard: 'https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc'
      - &redis_queue_size_crit
        alert: RedisQueueSize
        expr:
          redis_queue_total{
              cluster="frqueue",
              queue=~"(contribution_tracking|payments_init|pending|refund)"}
            > 2000
        for: 5m
        labels:
          severity: critical
          team: 'fr-tech'
        annotations:
          description: 'Critical redis queue size'
          summary: "Redis Queue {{ $labels.queue }} is Critical: [{{ $value }}]"
          dashboard: 'https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc'

Does this make sense?

As far as the PfwCoreBGPDown alert, when reading through the config I think we are getting email alerts due to this rule. When we have the new groups set up, maybe we can have that route to our paging level.

In T367370#11361652, @Dwisehaupt wrote:

@fgiunchedi We (fr-tech) are getting close to live testing with some alerts. We have started to build a set of alerts and are firing them off to our local alertmanager instance that will just send us email. With a config change, we could start pointing those at the production alertmanager instance.

We think the next logical step would be to set up the contact groups (team?) within alertmanager so that we can get them routed correctly via email/irc/splunk on-call. We want to make sure that we tag our alerts properly so that we don't cause issues for other folks. Is this something you can assist us with/point us in the right direction for?

Hi folks, I'm working on updating the Grafana alerts Wikitech section. It's still a WIP but I'd greatly appreciate your feedback:

Hi everyone,

In T406689#11277602, @Peter wrote:

#api-alerts channel

Ah cool, didn't know, I'll try that out, thank you!

In T406689#11270487, @Peter wrote:

Hi @andrea.denisse wanted to check if you know happened here, is it an upgrade that changed it? Please let me know if there's anything I can do to help!

I've finished the patch and the tests, now measuring by all of the clusters meaning all the responses are accounted for when measuring the percentages and the alerts would trigger as soon as those conditions are met. https://gerrit.wikimedia.org/r/c/operations/alerts/+/1192183/

Hi folks,

In T364622#11210082, @MoritzMuehlenhoff wrote:

In T364622#11208716, @andrea.denisse wrote:

Hi @MoritzMuehlenhoff and @Dzahn , we have an alert for the Puppet CA certificate for kibana.discovery.wmnet about to expire however, that certificate is no longer required as the services that use it were migrated to cfssl.
I've been unable to find that cert in the locations specified in the task, do you know where could it be located?

This certificate was created on the CA operated by the Puppet 5 servers (which is where we managed pretty much all internal certs before cfssl was introduced). You can remove it by logging into puppetmaster1001.eqiad.wmnet and running

sudo puppet cert clean kibana.discovery.wmnet

The alert should then auto-resolve a bit later.

Hi @MoritzMuehlenhoff and @Dzahn , we have an alert for the Puppet CA certificate for kibana.discovery.wmnet about to expire however, that certificate is no longer required as the services that use it were migrated to cfssl.

Hi folks, I wrote the documentation on using this to Wikitech. https://wikitech.wikimedia.org/wiki/Alertmanager#Sending_alerts_to_Slack

Hi folks, I’ve updated the Wikitech documentation for this feature. I’d really appreciate your feedback: https://wikitech.wikimedia.org/wiki/Alertmanager#Sending_alerts_to_Slack

Hi folks,

Hi folks, Grafana 12.1.1 introduced a couple of features regarding alerting,

In T402247#11096685, @andrea.denisse wrote:

In T402247#11096653, @andrea.denisse wrote:

I think that the drive is failing:

sudo dmesg | grep -i 'error\|fail\|ata':

[37968478.484217] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968478.495312] XFS: metadata IO error: 13 callbacks suppressed
[37968478.495415] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968478.646386] sd 0:2:5:0: [sdg] tag#345 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.690658] sd 0:2:5:0: [sdg] tag#804 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.714318] sd 0:2:5:0: [sdg] tag#806 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.728083] sd 0:2:5:0: [sdg] tag#807 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.744146] sd 0:2:5:0: [sdg] tag#808 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.760181] sd 0:2:5:0: [sdg] tag#810 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.760209] sd 0:2:5:0: [sdg] tag#810 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968478.760218] sd 0:2:5:0: [sdg] tag#810 Sense Key : Medium Error [current]
[37968478.760237] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968478.771419] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968478.783012] sd 0:2:5:0: [sdg] tag#817 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.796246] sd 0:2:5:0: [sdg] tag#772 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.812163] sd 0:2:5:0: [sdg] tag#779 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.828086] sd 0:2:5:0: [sdg] tag#782 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.848122] sd 0:2:5:0: [sdg] tag#787 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.868260] sd 0:2:5:0: [sdg] tag#791 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.868277] sd 0:2:5:0: [sdg] tag#791 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968478.868282] sd 0:2:5:0: [sdg] tag#791 Sense Key : Medium Error [current]
[37968478.868295] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968478.889217] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968478.900866] sd 0:2:5:0: [sdg] tag#330 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.920091] sd 0:2:5:0: [sdg] tag#331 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.940089] sd 0:2:5:0: [sdg] tag#332 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.964070] sd 0:2:5:0: [sdg] tag#334 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.984109] sd 0:2:5:0: [sdg] tag#337 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.004076] sd 0:2:5:0: [sdg] tag#341 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.004097] sd 0:2:5:0: [sdg] tag#341 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968479.004104] sd 0:2:5:0: [sdg] tag#341 Sense Key : Medium Error [current]
[37968479.004124] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968479.015325] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968479.146415] sd 0:2:5:0: [sdg] tag#10 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.160143] sd 0:2:5:0: [sdg] tag#11 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.176058] sd 0:2:5:0: [sdg] tag#12 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.192162] sd 0:2:5:0: [sdg] tag#13 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.208110] sd 0:2:5:0: [sdg] tag#14 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.224069] sd 0:2:5:0: [sdg] tag#15 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.224084] sd 0:2:5:0: [sdg] tag#15 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968479.224088] sd 0:2:5:0: [sdg] tag#15 Sense Key : Medium Error [current]
[37968479.224102] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968479.235286] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5

Looking further the drive in slot 3 shows several reallocated sectors:

==== Checking Slot 3 ==== Device Model: TOSHIBA MG06ACA800EY Serial Number: 81U0A02YF1QF SMART overall-health self-assessment test result: PASSED 5 Reallocated_Sector_Ct 0x0033 096 096 010 Pre-fail Always - 416 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0

I left a smartctl test running that will complete after Tue Aug 19 13:36:35 2025 UTC.

While investigating T402247 I left a smartctl test running for drive 3 (which is the one I suspect was failing due to the high number of reallocated sectors), here are the results.

In T383309#11095789, @andrea.denisse wrote:

In T383309#11093995, @tappof wrote:

@andrea.denisse Do you still have any debug logs available? I’m just curious...

Hi Tiziano, I don't have any debug logs. I captured and analyzed them in the host so the logs didn't leave the prod infra but I'll enable debug logging again to share them with the upstream maintainers as they would like to see the logs headers.

My plan is to enable debug logging and to share a sample of sanitized logs with the rsyslog maintainers for their advice. I can leave the file on the host if you'd like to analyze it, any findings you make could be pretty useful to further understanding or solving the issue.

In T402247#11099398, @Eevans wrote:

rsyslog is back up and running after clearing the queue (/var/spool/rsyslog/*), which apparently was corrupted.

In T402247#11096653, @andrea.denisse wrote:

I think that the drive is failing:

sudo dmesg | grep -i 'error\|fail\|ata':

[37968478.484217] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968478.495312] XFS: metadata IO error: 13 callbacks suppressed
[37968478.495415] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968478.646386] sd 0:2:5:0: [sdg] tag#345 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.690658] sd 0:2:5:0: [sdg] tag#804 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.714318] sd 0:2:5:0: [sdg] tag#806 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.728083] sd 0:2:5:0: [sdg] tag#807 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.744146] sd 0:2:5:0: [sdg] tag#808 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.760181] sd 0:2:5:0: [sdg] tag#810 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.760209] sd 0:2:5:0: [sdg] tag#810 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968478.760218] sd 0:2:5:0: [sdg] tag#810 Sense Key : Medium Error [current]
[37968478.760237] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968478.771419] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968478.783012] sd 0:2:5:0: [sdg] tag#817 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.796246] sd 0:2:5:0: [sdg] tag#772 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.812163] sd 0:2:5:0: [sdg] tag#779 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.828086] sd 0:2:5:0: [sdg] tag#782 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.848122] sd 0:2:5:0: [sdg] tag#787 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.868260] sd 0:2:5:0: [sdg] tag#791 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.868277] sd 0:2:5:0: [sdg] tag#791 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968478.868282] sd 0:2:5:0: [sdg] tag#791 Sense Key : Medium Error [current]
[37968478.868295] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968478.889217] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968478.900866] sd 0:2:5:0: [sdg] tag#330 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.920091] sd 0:2:5:0: [sdg] tag#331 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.940089] sd 0:2:5:0: [sdg] tag#332 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.964070] sd 0:2:5:0: [sdg] tag#334 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.984109] sd 0:2:5:0: [sdg] tag#337 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.004076] sd 0:2:5:0: [sdg] tag#341 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.004097] sd 0:2:5:0: [sdg] tag#341 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968479.004104] sd 0:2:5:0: [sdg] tag#341 Sense Key : Medium Error [current]
[37968479.004124] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968479.015325] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968479.146415] sd 0:2:5:0: [sdg] tag#10 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.160143] sd 0:2:5:0: [sdg] tag#11 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.176058] sd 0:2:5:0: [sdg] tag#12 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.192162] sd 0:2:5:0: [sdg] tag#13 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.208110] sd 0:2:5:0: [sdg] tag#14 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.224069] sd 0:2:5:0: [sdg] tag#15 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.224084] sd 0:2:5:0: [sdg] tag#15 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968479.224088] sd 0:2:5:0: [sdg] tag#15 Sense Key : Medium Error [current]
[37968479.224102] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968479.235286] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5

I think that the drive is failing:

I took a look at this and it seems like all of the alerts use the default alerting policy which delivers the notifications to Alertmanager and there are 6 actively alerting notifications that use that policy but I was unable to see them on Karma.

In T401730#11091376, @Mooeypoo wrote:

In T401730#11088178, @andrea.denisse wrote:

@Mooeypoo , do you know if these alerts are already present in Prometheus?

We don't yet have any alerts set up, but if we need to we can definitely come up with one we'd need and can test with,

In T401730#11090663, @andrea.denisse wrote:

In T401730#11090237, @colewhite wrote:

In T401730#11088178, @andrea.denisse wrote:

@colewhite, are there any downsides to using the webhook method instead of email?

I believe this isn't possible because the grafana hosts cannot connect outside the production network. They can send emails through our internal mail servers, though.

But the Alertmanager hosts can connect outside of production so I think that the webhook can be used. The Alertmanager hosts already communicate with SplunkOnCall along with the Prometheus dead man switch.

As for the Grafana alerts, I think that they can be routed to Alertamanger and then Alertmanager could send the alerts to Slack using the webhook.

In T383309#11093995, @tappof wrote:

@andrea.denisse Do you still have any debug logs available? I’m just curious...

Hi folks,

In T401730#11090237, @colewhite wrote:

In T401730#11088178, @andrea.denisse wrote:

@colewhite, are there any downsides to using the webhook method instead of email?

I believe this isn't possible because the grafana hosts cannot connect outside the production network. They can send emails through our internal mail servers, though.

Hi @hnowlan, I noticed the parent task is T401908. Is the goal here to ingest Grafana alerts into Alertmanager before sending them to Slack, or to route the alerts Alertmanager already receives into Slack channels?

Looking at our Alertmanager configuration, we currently send Slack notifications by creating an email address for a channel and sending alerts to it via email. However, Alertmanager can also send alerts directly to Slack using a webhook.

Hi Filippo, thanks for reviewing this! I had a question about this part: