Maniphest T327960

sessionstore: alert on rate of status 500 responses
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Eevans
	Jan 25 2023, 8:50 PM

Tags

Referenced Files

None

Subscribers

Tokens

"Orange Medal" token, awarded by BCornwall.

Description

Notifications based on the rate of HTTP status 500 responses from Kask would be an excellent indicator of user-facing problems.

See also:

https://wikitech.wikimedia.org/wiki/Incidents/2023-01-24_sessionstore_quorum_issues (incident which would have benefited)
T320401: Alert on Kask error rate (related/similar)

Details

	Subject	Repo	Branch	Lines +/-
	data-persistence: alert on elevated sessions store error rate (5xx)	operations/alerts	master	+76 -0

Customize query in gerrit

Related Objects

Mentioned In: T320401: Alert on Kask error rate
Mentioned Here: T320401: Alert on Kask error rate

Event Timeline

Eevans created this task.Jan 25 2023, 8:50 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 25 2023, 8:50 PM

Eevans renamed this task from sessionstore: alert on rate of 500 status to sessionstore: alert on rate of status 500 responses.Jan 25 2023, 8:50 PM

Eevans triaged this task as Medium priority.

Eevans mentioned this in T320401: Alert on Kask error rate.Jan 25 2023, 9:07 PM

BCornwall subscribed.Jan 25 2023, 10:37 PM

JJMC89 updated the task description. (Show Details)Jan 26 2023, 5:04 AM

Clement_Goubert edited projects, added serviceops; removed SRE.Jan 26 2023, 10:23 AM

Clement_Goubert moved this task from Incoming 🐫 to 🛎 Services & Oids on the serviceops board.Jan 26 2023, 12:31 PM

Change 893538 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/alerts@master] data-persistence: alert on elevated sessions store error rate (5xx)

https://gerrit.wikimedia.org/r/893538

gerritbot added a project: Patch-For-Review.Mar 1 2023, 9:37 PM

Change 893538 merged by Eevans:

[operations/alerts@master] data-persistence: alert on elevated sessions store error rate (5xx)

https://gerrit.wikimedia.org/r/893538

Maintenance_bot removed a project: Patch-For-Review.Mar 8 2023, 6:10 PM

BCornwall awarded a token.Mar 9 2023, 3:39 PM

This is complete; Closing