wdqs: alert on ratio of failed queries increase
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	gmodena
	Jan 12 2026, 9:31 AM

Description

As a developer and operator of WDQS I would like to receive an alert at warning level when the ratio of failed queries increases compared to baseline.

While this metric is not entirely part of our SLIs (the SLO considers 403 and 419 acceptable responses), it would be useful to be aware of changes in traffic patterns to proactively monitor the services.

Note: the current implementation of "failed queries" in Grafana includes both 4xx and 5xx. Those are different failure scenarios, we should consider reporting and alerting them separately. In particular, we are interested in tracking timeouts.

we have a refined definition of "failed query", aligned with SLO.
we have defined a quantifiable baseline of expected "failed queries" ratio.
we have an alert (warning) for increased ratio of failed queries (and/or timeouts).

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
wdp: blazegraph: increase alert threshold for 5xx	operations/alerts	master	+1 -1
wdqs: detune BlazegraphFailedServerRatioIncrease	operations/alerts	master	+6 -6
blazegraph: alert on ratio of failed queries increase	operations/alerts	master	+134 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T414782 Hypothesis WE2.5.1: setup triple stores in eqiad
		Resolved		trueg	T414306 wdqs: alert on ratio of failed queries increase

Event Timeline

gmodena created this task.Jan 12 2026, 9:31 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 12 2026, 9:31 AM

gmodena renamed this task from wdqs: alert on ratio of failed queries increase to [NEEDS GROOMING] wdqs: alert on ratio of failed queries increase.Jan 12 2026, 9:38 AM

Maintenance_bot added a project: Wikidata.Jan 12 2026, 10:29 AM

BTracy-WMF subscribed.Jan 12 2026, 2:32 PM

trueg added a project: OKR-Work.Jan 13 2026, 2:46 PM

Note to self:

runbook update
mention the issue that we cannot distinguish between failures on proxy level and blazegraph level

trueg updated the task description. (Show Details)Jan 14 2026, 3:22 PM

trueg renamed this task from [NEEDS GROOMING] wdqs: alert on ratio of failed queries increase to wdqs: alert on ratio of failed queries increase.Jan 14 2026, 3:26 PM

trueg changed the task status from Open to In Progress.

trueg claimed this task.

Change #1227364 had a related patch set uploaded (by Trueg; author: Trueg):

[operations/alerts@master] blazegraph: alert on ratio of failed queries increase

https://gerrit.wikimedia.org/r/1227364

gerritbot added a project: Patch-For-Review.Jan 15 2026, 3:52 PM

gmodena added a parent task: T414782: Hypothesis WE2.5.1: setup triple stores in eqiad.Jan 16 2026, 10:36 AM

@gmodena I am not entirely sure what is meant by "aligned with SLO" when it comes to the "definition of "failed query"".

Change #1227364 merged by jenkins-bot:

[operations/alerts@master] blazegraph: alert on ratio of failed queries increase

https://gerrit.wikimedia.org/r/1227364

In T414306#11529589, @trueg wrote:

f/up from a chat we had earlier:

@gmodena I am not entirely sure what is meant by "aligned with SLO" when it comes to the "definition of "failed query"".

The SLO describes 403 and 419 as acceptable responses. To align, we decided not to alert on 403 and 419 (they don't count towards WDQS' error budget).

Maintenance_bot removed a project: Patch-For-Review.Jan 19 2026, 2:32 PM

trueg triaged this task as Medium priority.Jan 20 2026, 9:50 AM

trueg edited projects, added Wikidata-Query-Service (Current Sprint); removed Wikidata-Query-Service.

trueg moved this task from Backlog to In Engineering on the Wikidata-Query-Service (Current Sprint) board.

trueg moved this task from In Engineering to In Code Review on the Wikidata-Query-Service (Current Sprint) board.Jan 20 2026, 3:34 PM

trueg closed this task as Resolved.Jan 29 2026, 10:47 AM

trueg reopened this task as In Progress.

trueg added a project: Wikidata Platform Team (Sprint 01 (2026/01/13)).

trueg moved this task from Backlog to Needs Sign-off on the Wikidata Platform Team (Sprint 01 (2026/01/13)) board.

DSantamaria removed a project: Wikidata-Query-Service (Current Sprint).Feb 4 2026, 12:48 PM

Change #1236852 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/alerts@master] wdqs: detune BlazegraphFailedServerRatioIncrease

https://gerrit.wikimedia.org/r/1236852

gerritbot added a project: Patch-For-Review.Feb 4 2026, 11:04 PM

Change #1236852 merged by jenkins-bot:

[operations/alerts@master] wdqs: detune BlazegraphFailedServerRatioIncrease

https://gerrit.wikimedia.org/r/1236852

We've tuned the alert slightly; a little over a third of the time we were seeing the alert fire for between 1-14 minutes and then resolve. So we've bumped from 30m to 45m to make the alert more actionable on the SRE side. Let us know if there's any issues with the change; for the timebeing I've merged the patch.

Maintenance_bot removed a project: Patch-For-Review.Feb 5 2026, 9:31 PM

thanks @RKemper . The alert is still a bit spammy. Taking a look now.

Change #1239088 had a related patch set uploaded (by Gmodena; author: Gmodena):

[operations/alerts@master] wdp: blazegraph: increase alert threshold for 5xx

https://gerrit.wikimedia.org/r/1239088

gerritbot added a project: Patch-For-Review.Thu, Feb 12, 12:51 PM

Change #1239088 merged by jenkins-bot:

[operations/alerts@master] wdp: blazegraph: increase alert threshold for 5xx

https://gerrit.wikimedia.org/r/1239088

Maintenance_bot removed a project: Patch-For-Review.Thu, Feb 12, 1:31 PM

BTracy-WMF closed this task as Resolved.Thu, Feb 26, 12:04 PM