Change Details

As a developer and operator of WDQS I would like to receive an alert at `warning` level when the [ratio of failed queries](https://grafana-rw.wikimedia.org/d/000000489/wikidata-query-service?forceLogin=true&from=now-30m&orgId=1&refresh=1m&timezone=utc&to=now&var-cluster_name=wdqs-legacy-full&var-graph_type=%289102%7C919%5B35%5D%29&viewPanel=panel-43) increases compared to baseline. While this metric is not entirely part of our SLIs (the SLO considers 403 and 419 acceptable responses), it would be useful to be aware of changes in traffic patterns to proactively monitor the services. **Note**: the current implementation of "failed queries" in Grafana includes both 4xx and 5xx. Those are different failure scenarios, we should consider reporting and alerting them separately. In particular, we are interested in tracking timeouts. AC - [] we have a refined definition of "failed query", aligned with SLO. - [] we have defined a quantifiable baseline of expected "failed queries" ratio. - [] we have an alert (`warning`) for increased ratio of failed queries (and/or timeouts).

As a developer and operator of WDQS I would like to receive an alert at `warning` level when the [ratio of failed queries](https://grafana-rw.wikimedia.org/d/000000489/wikidata-query-service?forceLogin=true&from=now-30m&orgId=1&refresh=1m&timezone=utc&to=now&var-cluster_name=wdqs-legacy-full&var-graph_type=%289102%7C919%5B35%5D%29&viewPanel=panel-43) increases compared to baseline. While this metric is not entirely part of our SLIs (the SLO considers 403 and 419 acceptable responses), it would be useful to be aware of changes in traffic patterns to proactively monitor the services. **Note**: the current implementation of "failed queries" in Grafana includes both 4xx and 5xx. Those are different failure scenarios, we should consider reporting and alerting them separately. In particular, we are interested in tracking timeouts. AC - [] we have a refined definition of "failed query", aligned with [[ https://wikitech.wikimedia.org/wiki/SLO/WDQS | SLO ]]. - [] we have defined a quantifiable baseline of expected "failed queries" ratio. - [] we have an alert (`warning`) for increased ratio of failed queries (and/or timeouts).

As a developer and operator of WDQS I would like to receive an alert at `warning` level when the [ratio of failed queries](https://grafana-rw.wikimedia.org/d/000000489/wikidata-query-service?forceLogin=true&from=now-30m&orgId=1&refresh=1m&timezone=utc&to=now&var-cluster_name=wdqs-legacy-full&var-graph_type=%289102%7C919%5B35%5D%29&viewPanel=panel-43) increases compared to baseline. While this metric is not entirely part of our SLIs (the SLO considers 403 and 419 acceptable responses), it would be useful to be aware of changes in traffic patterns to proactively monitor the services. **Note**: the current implementation of "failed queries" in Grafana includes both 4xx and 5xx. Those are different failure scenarios, we should consider reporting and alerting them separately. In particular, we are interested in tracking timeouts. AC - [] we have a refined definition of "failed query", aligned with SLO[[ https://wikitech.wikimedia.org/wiki/SLO/WDQS | SLO ]]. - [] we have defined a quantifiable baseline of expected "failed queries" ratio. - [] we have an alert (`warning`) for increased ratio of failed queries (and/or timeouts).