Page MenuHomePhabricator
Create Task
Maniphest T345358

DatasourceError alerts emitted by jinxer are unhelpful
Closed, ResolvedPublic
Actions

Description

12:04:45	<jinxer-wm>	(DatasourceError) firing: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError
12:19:45	<jinxer-wm>	(DatasourceError) resolved: <no value>   - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError

Jinxer at least indicate where the alert came from without having to resort to go digging through the logs.

Details

SubjectRepoBranchLines +/-
alertmanager: add link to DatasourceError runbookoperations/puppetproduction+1 -1
alertmanager: emit helpful info for DatasourceError alertsoperations/puppetproduction+4 -0
logstash: send generatorURL to labelsoperations/puppetproduction+3 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

colewhite created this task.Aug 31 2023, 1:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 31 2023, 1:58 PM
colewhite added a comment.Aug 31 2023, 1:59 PM

Initial proposal: (DatasourceError) firing: {{ rulename }} - {{ generatorURL }} - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError

gerritbot added a comment.Aug 31 2023, 2:19 PM

Change 953492 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] alertmanager: emit helpful info for DatasourceError alerts

https://gerrit.wikimedia.org/r/953492

gerritbot added a project: Patch-For-Review.Aug 31 2023, 2:19 PM

Change 953493 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: send generatorURL to labels

https://gerrit.wikimedia.org/r/953493

gerritbot added a comment.Aug 31 2023, 2:55 PM

Change 953493 merged by Cwhite:

[operations/puppet@production] logstash: send generatorURL to labels

https://gerrit.wikimedia.org/r/953493

gerritbot added a comment.Aug 31 2023, 4:49 PM

Change 953492 merged by Cwhite:

[operations/puppet@production] alertmanager: emit helpful info for DatasourceError alerts

https://gerrit.wikimedia.org/r/953492

Maintenance_bot removed a project: Patch-For-Review.Aug 31 2023, 5:10 PM
gerritbot added a comment.Aug 31 2023, 5:16 PM

Change 953495 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] alertmanager: add link to DatasourceError runbook

https://gerrit.wikimedia.org/r/953495

gerritbot added a project: Patch-For-Review.Aug 31 2023, 5:16 PM
colewhite changed the task status from Open to In Progress.Aug 31 2023, 5:17 PM
colewhite triaged this task as Medium priority.
colewhite added a parent task: T317887: Upgrade to Grafana 9.Aug 31 2023, 5:23 PM
gerritbot added a comment.Sep 12 2023, 8:37 PM

Change 953495 merged by Cwhite:

[operations/puppet@production] alertmanager: add link to DatasourceError runbook

https://gerrit.wikimedia.org/r/953495

Maintenance_bot removed a project: Patch-For-Review.Sep 12 2023, 9:10 PM
colewhite closed this task as Resolved.Sep 12 2023, 9:13 PM
colewhite added a project: SRE Observability (FY2023/2024-Q1).
colewhite moved this task from Inbox to Done on the SRE Observability (FY2023/2024-Q1) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits