The DatasourceNoData and DatasourceError alerts need amending, but are supressed from firing due to T317887#8783039. From the Performance team responsibilities sheet, QTE now owns these alerts.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | colewhite | T317887 Upgrade to Grafana 9 | |||
| Open | Peter | T344961 "browsertime-alerts" and "webpagereplay" alerts: DatasourceNoData and DatasourceError silenced |
We’re also seeing DatasourceError alerts on several Wikidata alerts, usually with the message “failed to build query 'A': database is locked”. Is that the same issue?
I'm back from vacation since some time, I'll have a look at them. Some of the alerts I think will be moved to the web team, but let me sync with them ASAP.
- Normal Speed Index Firefox Desktop ALERT alert
Hmm that was a really old alert that was broken. I've fixed the dashboard and removed that alert for now.
The rest of the alerts "increased transfer size" are correct and we need to understand the root cause. The other alerts could be caused by those, I'll check that tomorrow.
I've increased the time span for all the alerts to make sure it do not hit "no data". Lets wait until I see that it works correctly before closing.
I just got an email alert relating to this today:
[FIRING:1] DatasourceError rweb (qbUqRRb4k "AlertManager","ReadingWeb","cxserver" Readers Web [Desktop] Navigation Timing: First paint (Moving Medians) critical)
https://grafana.wikimedia.org/alerting/grafana/qbUqRRb4k/view
Note that DatasourceNoData is still regularly received on Graphite-based metric alerts as recently as two days ago.
10 Feb 2024: [FIRING:1] DatasourceError perf (JSAFSCJVk "AlertManager" https://grafana.wikimedia.org/d/000000326/navigation-timing-alerts …)
This suggests to me that, contrary to T317887, these either do not have a single root cause, or that the root cause remains at least partially unresolved.
And Prometheus-based alerts as well, from earlier today:
14 Feb 2024: [FIRING:1] DatasourceNoData mediawiki-platform (SvAKSjJVz "AlertManager" https://grafana.wikimedia.org/d/000000402/resourceloader-alerts 000000026 https://grafana.wikimedia.org/d/000000066/resourceloader "resourceloader INM Satisfaction")
https://grafana.wikimedia.org/alerting/grafana/SvAKSjJVz/view