Page MenuHomePhabricator
Create Task
Maniphest T323064

Create WDQS Uptime SLO dashboard in Grizzly
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

We use Grizzly to create org-wide SLO dashboards.

AC:

Details

SubjectRepoBranchLines +/-
wdqs: make uptime sli a %operations/grafana-grizzlymaster+1 -1
wdqs: fix request request error ratio sli paneoperations/grafana-grizzlymaster+1 -1
wdqs: use pre-computed wdqs recording rulesoperations/grafana-grizzlymaster+3 -3
wdqs: add recording rule for req success ratiooperations/puppetproduction+14 -2
wdqs: add recording rule for req success ratiooperations/puppetproduction+14 -2
add grizzly dashboard for WDQS uptimeoperations/grafana-grizzlymaster+11 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

RKemper created this task.Nov 14 2022, 7:50 PM
Gehel triaged this task as High priority.Nov 21 2022, 4:26 PM
Gehel moved this task from Incoming to Current work on the Wikidata-Query-Service board.
Gehel set the point value for this task to 3.Nov 21 2022, 4:45 PM
Gehel moved this task from Incoming to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board.
gerritbot added a comment.Nov 30 2022, 7:47 AM

Change 862178 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/grafana-grizzly@master] [WIP] add grizzly dashboard for WDQS uptime

https://gerrit.wikimedia.org/r/862178

gerritbot added a project: Patch-For-Review.Nov 30 2022, 7:47 AM
Gehel moved this task from Ready for Dev -- SRE/Ops to Blocked/Waiting on the Discovery-Search (Current work) board.Dec 1 2022, 7:33 PM
Gehel moved this task from Blocked/Waiting to In Progress on the Discovery-Search (Current work) board.
gerritbot added a comment.Dec 8 2022, 8:10 PM

Change 862178 merged by Ryan Kemper:

[operations/grafana-grizzly@master] add grizzly dashboard for WDQS uptime

https://gerrit.wikimedia.org/r/862178

Stashbot added a comment.Dec 8 2022, 8:17 PM

Mentioned in SAL (#wikimedia-operations) [2022-12-08T20:17:23Z] <ryankemper> T323064 Merged https://gerrit.wikimedia.org/r/c/operations/grafana-grizzly/+/862178 and deployed new dashboard, visible here: https://grafana.wikimedia.org/d/slo-wdqs-tmpl/wdqs-slos-grizzly-template?orgId=1

RKemper added a comment.Dec 8 2022, 8:17 PM
ryankemper@grafana1002:/srv/grafana-grizzly$ grr diff slo_dashboards.jsonnet
Dashboard/slo-logstash-tmpl no differences
Dashboard/slo-trafficserver-tmpl no differences
Dashboard/slo-varnish-tmpl no differences
Dashboard/slo-wdqs-tmpl not present in Dashboard
Dashboard/slo-apigw no differences
Dashboard/slo-etcd-tmpl no differences
Dashboard/slo-haproxy-tmpl no differences
ryankemper@grafana1002:/srv/grafana-grizzly$ grr apply slo_dashboards.jsonnet
Dashboard/slo-logstash-tmpl no differences
Dashboard/slo-trafficserver-tmpl no differences
Dashboard/slo-varnish-tmpl no differences
Dashboard/slo-wdqs-tmpl added
Dashboard/slo-apigw no differences
Dashboard/slo-etcd-tmpl no differences
Dashboard/slo-haproxy-tmpl no differences
RKemper renamed this task from Create WDQS Uptime SLO && WDQS/WCQS update lag SLO dashboards in Grizzly to Create WDQS Uptime SLO dashboard in Grizzly.Dec 8 2022, 8:18 PM
RKemper updated the task description. (Show Details)
RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.
RKemper updated the task description. (Show Details)
Maintenance_bot removed a project: Patch-For-Review.Dec 8 2022, 8:31 PM
Gehel moved this task from Needs Reporting to Needs review on the Discovery-Search (Current work) board.Dec 9 2022, 7:57 AM

Looking at https://grafana.wikimedia.org/d/slo-wdqs-tmpl/wdqs-slos-grizzly-template?orgId=1&var-datasource=thanos&var-site=All&var-cluster=All, I only see data for "Request Error Budget Remaining". The other graphs don't seem to be timing out, so I suspect there is an issue with the queries. I have not investigated further.

Moving this ticket back to needs review.

gerritbot added a comment.Dec 13 2022, 7:56 PM

Change 867695 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/grafana-grizzly@master] wdqs: fix request request error ratio sli pane

https://gerrit.wikimedia.org/r/867695

gerritbot added a project: Patch-For-Review.Dec 13 2022, 7:56 PM
Gehel moved this task from Needs review to In Progress on the Discovery-Search (Current work) board.Jan 9 2023, 4:05 PM
gerritbot added a comment.Jan 12 2023, 4:49 PM

Change 879599 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] [WIP] wdqs: add recording rule for req success ratio

https://gerrit.wikimedia.org/r/879599

gerritbot added a comment.Jan 12 2023, 5:16 PM

Change 879606 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/grafana-grizzly@master] [WIP] wdqs: use pre-computed wdqs recording rules

https://gerrit.wikimedia.org/r/879606

gerritbot added a comment.Jan 24 2023, 7:40 PM

Change 879599 merged by Ryan Kemper:

[operations/puppet@production] wdqs: add recording rule for req success ratio

https://gerrit.wikimedia.org/r/879599

Gehel added a comment.Jan 25 2023, 9:48 AM

above patch was reverted in https://gerrit.wikimedia.org/r/c/operations/puppet/+/883223/

gerritbot added a comment.Jan 25 2023, 4:37 PM

Change 883610 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/puppet@production] wdqs: add recording rule for req success ratio

https://gerrit.wikimedia.org/r/883610

gerritbot added a comment.Jan 26 2023, 8:12 PM

Change 883610 merged by Ryan Kemper:

[operations/puppet@production] wdqs: add recording rule for req success ratio

https://gerrit.wikimedia.org/r/883610

fgiunchedi mentioned this in T328306: Thanos rule evaluation alerts for service_slis.Jan 30 2023, 2:11 PM
gerritbot added a comment.Feb 28 2023, 8:36 PM

Change 879606 merged by Ryan Kemper:

[operations/grafana-grizzly@master] wdqs: use pre-computed wdqs recording rules

https://gerrit.wikimedia.org/r/879606

gerritbot added a comment.Feb 28 2023, 8:37 PM

Change 867695 abandoned by Ryan Kemper:

[operations/grafana-grizzly@master] wdqs: fix request request error ratio sli pane

Reason:

obsoleted by https://gerrit.wikimedia.org/r/c/operations/grafana-grizzly/+/879606

https://gerrit.wikimedia.org/r/867695

Maintenance_bot removed a project: Patch-For-Review.Feb 28 2023, 9:12 PM
RKemper moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board.Mar 6 2023, 4:26 PM
Gehel closed this task as Resolved.Mar 10 2023, 2:14 PM
gerritbot added a comment.Apr 27 2023, 6:57 PM

Change 912944 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/grafana-grizzly@master] wdqs: make uptime sli a %

https://gerrit.wikimedia.org/r/912944

gerritbot added a project: Patch-For-Review.Apr 27 2023, 6:57 PM
gerritbot added a comment.Apr 27 2023, 7:04 PM

Change 912944 merged by Ryan Kemper:

[operations/grafana-grizzly@master] wdqs: make uptime sli a %

https://gerrit.wikimedia.org/r/912944

Maintenance_bot removed a project: Patch-For-Review.Apr 27 2023, 7:10 PM
RKemper added a comment.May 9 2023, 7:59 PM

Forgot to link patch but here's the (hopefully final) grizzly patch to get this where we want it: https://gerrit.wikimedia.org/r/c/operations/grafana-grizzly/+/917938

(patch already merged & deployed)

Gehel closed subtask T328306: Thanos rule evaluation alerts for service_slis as Resolved.Jun 2 2023, 9:43 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL