In T411436#11426873, @andrea.denisse wrote:

In T411436#11425341, @SEgt-WMF wrote:

In case it is useful: the MediaWiki page @Rmaung pointed out was the first thing that came up for me when I googled "Request Access Superset Wikimedia", so it might be good to update that one too? In any case, I'm super happy to learn about the proper process, so thank you, @Aklapper!

Hi Silvia, could you please update the task with the fields from the https://phabricator.wikimedia.org/maniphest/task/edit/form/8/ form?

Hi folks, if I understand correctly the access request can't be fulfilled because the requested type of access doesn't exist.

Hi folks, the patch for this task is merged. Can we close it as resolved?

Closing as resolved, feel free to reopen if there's anything else I can assist with.

Closing as resolved, please let me know if there's anything else I can assist with.

Closing as resolved, please let me know if there's anything else I can assist with.

Closing as resolved, please let me know if there's anything else I can assist with.

I wrote to Zoe directly to confirm of this request.

Was collaboration-services tag removed? This tag showed up on the Clinic Duty dashboard.

In T411436#11425341, @SEgt-WMF wrote:

In case it is useful: the MediaWiki page @Rmaung pointed out was the first thing that came up for me when I googled "Request Access Superset Wikimedia", so it might be good to update that one too? In any case, I'm super happy to learn about the proper process, so thank you, @Aklapper!

Hi @KFrancis, I was unable to find @medelius on the NDA spreadsheet, could you please help me to confirm their NDA status?

Hi @VPuffetMichel , do you approve this request?

Hi Raine, this is on the clinic duty dashboard.

Hi Hugh, this is on the clinic duty dashboard.

In T367370#11368418, @Dwisehaupt wrote:

@andrea.denisse Thanks for the feedback. We want to mimic what we currently have for the icinga alerting groups. I think I have done that with this commit (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1204648). It goes a little further to split out fr-tech and fr-tech-ops since we have some hardware/OS level alerts that the whole fr-tech team doesn't need to have.

Our alert rules will live in fundraising and fire from our prometheus server. Given that, I believe we can reference the team in our rules to make sure that they get properly routed. Here is an example of one our our rules we have been testing locally:

---

groups:

  - name: queue-alerts

    rules:

      - &redis_queue_size_warn
        alert: RedisQueueSize
        expr:
          redis_queue_total{
              cluster="frqueue",
              queue=~"(contribution_tracking|payments_init|pending|refund)"}
            > 1500
        for: 5m
        labels:
          severity: warning
          team: 'fr-tech'
        annotations:
          description: 'High redis queue size'
          summary: "Redis Queue {{ $labels.queue }} is high: [{{ $value }}]"
          dashboard: 'https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc'
      - &redis_queue_size_crit
        alert: RedisQueueSize
        expr:
          redis_queue_total{
              cluster="frqueue",
              queue=~"(contribution_tracking|payments_init|pending|refund)"}
            > 2000
        for: 5m
        labels:
          severity: critical
          team: 'fr-tech'
        annotations:
          description: 'Critical redis queue size'
          summary: "Redis Queue {{ $labels.queue }} is Critical: [{{ $value }}]"
          dashboard: 'https://frmon.wikimedia.org/d/R5m3iU1Wk/queue?orgId=1&from=now-24h&to=now&timezone=utc'

Does this make sense?

As far as the PfwCoreBGPDown alert, when reading through the config I think we are getting email alerts due to this rule. When we have the new groups set up, maybe we can have that route to our paging level.

In T367370#11361652, @Dwisehaupt wrote:

@fgiunchedi We (fr-tech) are getting close to live testing with some alerts. We have started to build a set of alerts and are firing them off to our local alertmanager instance that will just send us email. With a config change, we could start pointing those at the production alertmanager instance.

We think the next logical step would be to set up the contact groups (team?) within alertmanager so that we can get them routed correctly via email/irc/splunk on-call. We want to make sure that we tag our alerts properly so that we don't cause issues for other folks. Is this something you can assist us with/point us in the right direction for?

Hi folks, I'm working on updating the Grafana alerts Wikitech section. It's still a WIP but I'd greatly appreciate your feedback:

Hi everyone,

In T406689#11277602, @Peter wrote:

#api-alerts channel

Ah cool, didn't know, I'll try that out, thank you!

In T406689#11270487, @Peter wrote:

Hi @andrea.denisse wanted to check if you know happened here, is it an upgrade that changed it? Please let me know if there's anything I can do to help!

I've finished the patch and the tests, now measuring by all of the clusters meaning all the responses are accounted for when measuring the percentages and the alerts would trigger as soon as those conditions are met. https://gerrit.wikimedia.org/r/c/operations/alerts/+/1192183/

Hi folks,

In T364622#11210082, @MoritzMuehlenhoff wrote:

In T364622#11208716, @andrea.denisse wrote:

Hi @MoritzMuehlenhoff and @Dzahn , we have an alert for the Puppet CA certificate for kibana.discovery.wmnet about to expire however, that certificate is no longer required as the services that use it were migrated to cfssl.
I've been unable to find that cert in the locations specified in the task, do you know where could it be located?

This certificate was created on the CA operated by the Puppet 5 servers (which is where we managed pretty much all internal certs before cfssl was introduced). You can remove it by logging into puppetmaster1001.eqiad.wmnet and running

sudo puppet cert clean kibana.discovery.wmnet

The alert should then auto-resolve a bit later.

Hi @MoritzMuehlenhoff and @Dzahn , we have an alert for the Puppet CA certificate for kibana.discovery.wmnet about to expire however, that certificate is no longer required as the services that use it were migrated to cfssl.

Hi folks, I wrote the documentation on using this to Wikitech. https://wikitech.wikimedia.org/wiki/Alertmanager#Sending_alerts_to_Slack

Hi folks, I’ve updated the Wikitech documentation for this feature. I’d really appreciate your feedback: https://wikitech.wikimedia.org/wiki/Alertmanager#Sending_alerts_to_Slack

Hi folks,

Hi folks, Grafana 12.1.1 introduced a couple of features regarding alerting,

In T402247#11096685, @andrea.denisse wrote:

In T402247#11096653, @andrea.denisse wrote:

I think that the drive is failing:

sudo dmesg | grep -i 'error\|fail\|ata':

[37968478.484217] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968478.495312] XFS: metadata IO error: 13 callbacks suppressed
[37968478.495415] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968478.646386] sd 0:2:5:0: [sdg] tag#345 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.690658] sd 0:2:5:0: [sdg] tag#804 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.714318] sd 0:2:5:0: [sdg] tag#806 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.728083] sd 0:2:5:0: [sdg] tag#807 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.744146] sd 0:2:5:0: [sdg] tag#808 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.760181] sd 0:2:5:0: [sdg] tag#810 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.760209] sd 0:2:5:0: [sdg] tag#810 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968478.760218] sd 0:2:5:0: [sdg] tag#810 Sense Key : Medium Error [current]
[37968478.760237] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968478.771419] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968478.783012] sd 0:2:5:0: [sdg] tag#817 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.796246] sd 0:2:5:0: [sdg] tag#772 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.812163] sd 0:2:5:0: [sdg] tag#779 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.828086] sd 0:2:5:0: [sdg] tag#782 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.848122] sd 0:2:5:0: [sdg] tag#787 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.868260] sd 0:2:5:0: [sdg] tag#791 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.868277] sd 0:2:5:0: [sdg] tag#791 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968478.868282] sd 0:2:5:0: [sdg] tag#791 Sense Key : Medium Error [current]
[37968478.868295] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968478.889217] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968478.900866] sd 0:2:5:0: [sdg] tag#330 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.920091] sd 0:2:5:0: [sdg] tag#331 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.940089] sd 0:2:5:0: [sdg] tag#332 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.964070] sd 0:2:5:0: [sdg] tag#334 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.984109] sd 0:2:5:0: [sdg] tag#337 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.004076] sd 0:2:5:0: [sdg] tag#341 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.004097] sd 0:2:5:0: [sdg] tag#341 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968479.004104] sd 0:2:5:0: [sdg] tag#341 Sense Key : Medium Error [current]
[37968479.004124] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968479.015325] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968479.146415] sd 0:2:5:0: [sdg] tag#10 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.160143] sd 0:2:5:0: [sdg] tag#11 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.176058] sd 0:2:5:0: [sdg] tag#12 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.192162] sd 0:2:5:0: [sdg] tag#13 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.208110] sd 0:2:5:0: [sdg] tag#14 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.224069] sd 0:2:5:0: [sdg] tag#15 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.224084] sd 0:2:5:0: [sdg] tag#15 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968479.224088] sd 0:2:5:0: [sdg] tag#15 Sense Key : Medium Error [current]
[37968479.224102] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968479.235286] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5

Looking further the drive in slot 3 shows several reallocated sectors:

==== Checking Slot 3 ==== Device Model: TOSHIBA MG06ACA800EY Serial Number: 81U0A02YF1QF SMART overall-health self-assessment test result: PASSED 5 Reallocated_Sector_Ct 0x0033 096 096 010 Pre-fail Always - 416 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0

I left a smartctl test running that will complete after Tue Aug 19 13:36:35 2025 UTC.

While investigating T402247 I left a smartctl test running for drive 3 (which is the one I suspect was failing due to the high number of reallocated sectors), here are the results.

In T383309#11095789, @andrea.denisse wrote:

In T383309#11093995, @tappof wrote:

@andrea.denisse Do you still have any debug logs available? I’m just curious...

Hi Tiziano, I don't have any debug logs. I captured and analyzed them in the host so the logs didn't leave the prod infra but I'll enable debug logging again to share them with the upstream maintainers as they would like to see the logs headers.

My plan is to enable debug logging and to share a sample of sanitized logs with the rsyslog maintainers for their advice. I can leave the file on the host if you'd like to analyze it, any findings you make could be pretty useful to further understanding or solving the issue.

In T402247#11099398, @Eevans wrote:

rsyslog is back up and running after clearing the queue (/var/spool/rsyslog/*), which apparently was corrupted.

In T402247#11096653, @andrea.denisse wrote:

I think that the drive is failing:

sudo dmesg | grep -i 'error\|fail\|ata':

[37968478.484217] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968478.495312] XFS: metadata IO error: 13 callbacks suppressed
[37968478.495415] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968478.646386] sd 0:2:5:0: [sdg] tag#345 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.690658] sd 0:2:5:0: [sdg] tag#804 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.714318] sd 0:2:5:0: [sdg] tag#806 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.728083] sd 0:2:5:0: [sdg] tag#807 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.744146] sd 0:2:5:0: [sdg] tag#808 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.760181] sd 0:2:5:0: [sdg] tag#810 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.760209] sd 0:2:5:0: [sdg] tag#810 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968478.760218] sd 0:2:5:0: [sdg] tag#810 Sense Key : Medium Error [current]
[37968478.760237] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968478.771419] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968478.783012] sd 0:2:5:0: [sdg] tag#817 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.796246] sd 0:2:5:0: [sdg] tag#772 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.812163] sd 0:2:5:0: [sdg] tag#779 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.828086] sd 0:2:5:0: [sdg] tag#782 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.848122] sd 0:2:5:0: [sdg] tag#787 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.868260] sd 0:2:5:0: [sdg] tag#791 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.868277] sd 0:2:5:0: [sdg] tag#791 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968478.868282] sd 0:2:5:0: [sdg] tag#791 Sense Key : Medium Error [current]
[37968478.868295] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968478.889217] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968478.900866] sd 0:2:5:0: [sdg] tag#330 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.920091] sd 0:2:5:0: [sdg] tag#331 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.940089] sd 0:2:5:0: [sdg] tag#332 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.964070] sd 0:2:5:0: [sdg] tag#334 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968478.984109] sd 0:2:5:0: [sdg] tag#337 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.004076] sd 0:2:5:0: [sdg] tag#341 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.004097] sd 0:2:5:0: [sdg] tag#341 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968479.004104] sd 0:2:5:0: [sdg] tag#341 Sense Key : Medium Error [current]
[37968479.004124] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968479.015325] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5
[37968479.146415] sd 0:2:5:0: [sdg] tag#10 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.160143] sd 0:2:5:0: [sdg] tag#11 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.176058] sd 0:2:5:0: [sdg] tag#12 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.192162] sd 0:2:5:0: [sdg] tag#13 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.208110] sd 0:2:5:0: [sdg] tag#14 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.224069] sd 0:2:5:0: [sdg] tag#15 BRCM Debug mfi stat 0x2d, data len requested/completed 0x4000/0x0
[37968479.224084] sd 0:2:5:0: [sdg] tag#15 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[37968479.224088] sd 0:2:5:0: [sdg] tag#15 Sense Key : Medium Error [current]
[37968479.224102] blk_update_request: I/O error, dev sdg, sector 348391680 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
[37968479.235286] XFS (sdg1): metadata I/O error in "xfs_imap_to_bp+0x61/0xb0 [xfs]" at daddr 0x14c40100 len 32 error 5

I think that the drive is failing:

I took a look at this and it seems like all of the alerts use the default alerting policy which delivers the notifications to Alertmanager and there are 6 actively alerting notifications that use that policy but I was unable to see them on Karma.

In T401730#11091376, @Mooeypoo wrote:

In T401730#11088178, @andrea.denisse wrote:

@Mooeypoo , do you know if these alerts are already present in Prometheus?

We don't yet have any alerts set up, but if we need to we can definitely come up with one we'd need and can test with,

In T401730#11090663, @andrea.denisse wrote:

In T401730#11090237, @colewhite wrote:

In T401730#11088178, @andrea.denisse wrote:

@colewhite, are there any downsides to using the webhook method instead of email?

I believe this isn't possible because the grafana hosts cannot connect outside the production network. They can send emails through our internal mail servers, though.

But the Alertmanager hosts can connect outside of production so I think that the webhook can be used. The Alertmanager hosts already communicate with SplunkOnCall along with the Prometheus dead man switch.

As for the Grafana alerts, I think that they can be routed to Alertamanger and then Alertmanager could send the alerts to Slack using the webhook.

In T383309#11093995, @tappof wrote:

@andrea.denisse Do you still have any debug logs available? I’m just curious...

Hi folks,

In T401730#11090237, @colewhite wrote:

In T401730#11088178, @andrea.denisse wrote:

@colewhite, are there any downsides to using the webhook method instead of email?

I believe this isn't possible because the grafana hosts cannot connect outside the production network. They can send emails through our internal mail servers, though.

Hi @hnowlan, I noticed the parent task is T401908. Is the goal here to ingest Grafana alerts into Alertmanager before sending them to Slack, or to route the alerts Alertmanager already receives into Slack channels?

Looking at our Alertmanager configuration, we currently send Slack notifications by creating an email address for a channel and sending alerts to it via email. However, Alertmanager can also send alerts directly to Slack using a webhook.

Hi Filippo, thanks for reviewing this! I had a question about this part:

In T393894#10860988, @Dreamy_Jazz wrote:

I still see the issue:

Hi @Peter, I recently upgraded our Grafana instances to v12.0.1, so the issue might already be resolved in this version.

This is still ongoing on Grafana v12.0.1. I'll explore clustered rendering further.

Hi @Dreamy_Jazz , I recently upgraded our Grafana instances to v12.0.1 and I think that this issue is fixed now.
Could you please take a look at it and let me know if the issue is still present for you?

Thanks for the updates, @tappof, the revised task description looks much clearer now!

Hi @Vgutierrez, I've upgraded both Grafana instances to v12.0.1 and I can no longer reproduce the issue. I'll close this task as resolved, feel free to re-open in case it's still happening for you.

In T394045#10857637, @Krinkle wrote:

That file linked above is a screenshot uploaded by you. Nobody else could see your screenshot in the task description, they saw {F59935904} as plain text instead, because the file was private (this is likely due to a drag-and-drop bug in Phab). As a Phab admin, I get a button to "fix" these with one click, which, when it offered me this, I clicked "Yes". That generated the above attachment log message.

I have no input on this task.

Hi @Krinkle, thanks for taking a look. Did you take that screenshot from the grafana-next instance? I can't reproduce the error anymore in that instance. The main grafana instance doesn't include the patch for this issue.