Elasticsearch: Alert on upstream errors for MW API
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bking
	Fri, Apr 26, 9:41 PM

Description

Creating this as a follow up task from this incident .

Data Platform SRE and Search Platform should be alerted when the connection error rate from Mediawiki app servers to search clusters goes above a certain percentage. See this Envoy Telemetry dashboard for an example of what we could/should be alerting on.

Creating this ticket to:

Create alerts
~~Confirm operation~~ Decided against deliberately triggering, as the recent incident and the promtool rules evaluation should be enough to give us confidence in the alert.

Details

Other Assignee: RKemper

	Subject	Repo	Branch	Lines +/-
	elastic: remove backend failure check	operations/puppet	production	+2 -14
	search-platform: monitor/alert on elastic request failures	operations/alerts	master	+35 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		bking	T363694 Post incident tasks: Search missing results/unavailable for some eqiad users
		Resolved		bking	T363609 Elasticsearch: Alert on upstream errors for MW API

Event Timeline

bking created this task.Fri, Apr 26, 9:41 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFri, Apr 26, 9:41 PM

bking added a parent task: T363694: Post incident tasks: Search missing results/unavailable for some eqiad users.Mon, Apr 29, 2:05 PM

Gehel triaged this task as High priority.Mon, Apr 29, 2:24 PM

Gehel moved this task from needs triage to Ops / SRE on the Discovery-Search board.

bking renamed this task from Elasticsearch: Alert on downstream errors to Elasticsearch: Alert on upstream errors for MW API.Mon, Apr 29, 4:49 PM

bking claimed this task.

bking updated Other Assignee, added: RKemper.

Change #1025453 had a related patch set uploaded (by Bking; author: Bking):

[operations/alerts@master] search-platform: monitoring/alert on upstream MW API errors

https://gerrit.wikimedia.org/r/1025453

gerritbot added a project: Patch-For-Review.Mon, Apr 29, 9:19 PM

bking moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.04.15 - 2024.05.05) board.Tue, Apr 30, 8:52 PM

Change #1025453 merged by jenkins-bot:

[operations/alerts@master] search-platform: monitor/alert on elastic request failures

https://gerrit.wikimedia.org/r/1025453

Maintenance_bot removed a project: Patch-For-Review.Thu, May 2, 7:30 PM

bking updated the task description. (Show Details)Fri, May 3, 1:59 PM

Per the above CR, alerts are now in place. As such, I'm closing out this ticket.

Change #1026950 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elastic: remove backend failure check

https://gerrit.wikimedia.org/r/1026950

gerritbot added a project: Patch-For-Review.Fri, May 3, 3:00 PM

Change #1026950 merged by Bking: