WDQS: Making BlazegraphFailedServerRatioIncrease alerts less sensitive
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bking
	Feb 3 2026, 4:49 PM

Description

I've noticed a lot of these alerts flapping in #wikidata-data-platform-alerts IRC lately. As they fire frequently and tend to clear within 10 minutes, I think there is an opportunity to make them less sensitive.

Creating this ticket to:

Make the alert less sensitive
Ensure Wikidata platform team is comfortable with this change, and discuss further if not.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	wdp: blazegraph: increase alert threshold for 5xx	operations/alerts	master	+1 -1
	wdqs: detune BlazegraphFailedServerRatioIncrease	operations/alerts	master	+6 -6

Customize query in gerrit

Related Objects

Mentioned Here: T414306: wdqs: alert on ratio of failed queries increase

Event Timeline

bking created this task.Feb 3 2026, 4:49 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 3 2026, 4:49 PM

bking renamed this task from WDQS: Detune BlazegraphFailedServerRatioIncrease alerts to WDQS: Making BlazegraphFailedServerRatioIncrease alerts less sensitive.Feb 3 2026, 6:30 PM

bking updated the task description. (Show Details)

Change #1236852 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/alerts@master] wdqs: detune BlazegraphFailedServerRatioIncrease

https://gerrit.wikimedia.org/r/1236852

gerritbot added a project: Patch-For-Review.Feb 5 2026, 8:41 PM

Change #1236852 merged by jenkins-bot:

[operations/alerts@master] wdqs: detune BlazegraphFailedServerRatioIncrease

https://gerrit.wikimedia.org/r/1236852

I'll update the Wikidata platform team on T414306 with this change. I think given it's a very simple change of 30m -> 45m, and the original intent of the alert is still preserved, there's no need to block this on getting formal approval. So I'll mark this resolved.

RKemper updated the task description. (Show Details)Feb 5 2026, 9:04 PM

Maintenance_bot removed a project: Patch-For-Review.Feb 5 2026, 9:31 PM

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.Feb 6 2026, 8:28 AM

Change #1239088 had a related patch set uploaded (by Gmodena; author: Gmodena):

[operations/alerts@master] wdp: blazegraph: increase alert threshold for 5xx

https://gerrit.wikimedia.org/r/1239088

gerritbot added a project: Patch-For-Review.Feb 12 2026, 12:51 PM

Change #1239088 merged by jenkins-bot:

[operations/alerts@master] wdp: blazegraph: increase alert threshold for 5xx

https://gerrit.wikimedia.org/r/1239088

Maintenance_bot removed a project: Patch-For-Review.Feb 12 2026, 1:31 PM