Page MenuHomePhabricator

Page on ATS backend errors relative to traffic
Open, Needs TriagePublic

Description

Today ATSBackendErrorsHigh alert pages on an absolute number of errors (3/s) although I think the severity/SLO of the alert depends on the traffic levels of the backend service itself. In other words 3 errors/s on a service doing 3k requests/s is far different in terms of impact on users and SRE oncall than on a service doing, say, 10x or 100x less traffic.

Thus I'd like to propose switching ATSBackendErrorsHigh to page based on service availability i.e. failures / all requests and decide said number. Please also note that we can consider a blend of services in this case, namely keep some services on fix thresholds and some others based on availability. Having a single paging policy however will likely be easier to understand and troubleshoot.

Details

Event Timeline

This alert was created by @Joe back in the day, adding him to the discussion :)

We talked about this in the SLO meeting today -- one possible approach is to keep ATSBackendErrorsHigh as a default policy, but keep a list of services to exclude because they have SLO-driven availability alerts (which are effectively the same, except with a thoughtfully chosen alert threshold). That way, over time fewer and fewer services are covered by the default alert.

That default alert could either be a fixed rate like today, or at some default fixed ratio -- so in that sense it's orthogonal to the proposal here, but maybe helps inform the decision.

This is still an issue: SRE got paged for 10 5xx req/s on swift which was doing 2k req/s at the time (esams). I very much doubt it is worth paging engineers on absolute number of errors, irrespective of service traffic

Thanks for the update @fgiunchedi. We should certainly move forward on this, one way or the other.

@hnowlan: Would appreciate some input from you, or someone on olly on what they think about this as well.

We will be discussing this tomorrow in the Traffic meeting as well and will follow up.

For the most part I think @fgiunchedi's original proposal of a ratio based on requests makes the most sense here. We already do some service exclusion, which could possibly be expanded if there are problem services.

Thank you for the feedback @hnowlan @ssingh ! re: exclusion list I'm also expecting that moving to ratio-based paging will carry more signal (i.e. we can shrink the exclusion list)

Conversely, alerting on ratio will also cover low-traffic services that are erroring out just below the threshold. As it was the case today for gitlab-replica-a where the ATSBackendErrorsHigh page came in at 9:45 while the service started erroring about an hour earlier.

2026-08-13-122349_3753x1653_scrot.png (3,753×1,653 px, 310 KB)

Today also a swift.discovery.wmnet page came in, although IMHO it should have been a thumbor page instead. Swift itself was fine compared to its traffic levels:

2026-08-13-123018_3773x1765_scrot.png (3,773×1,765 px, 328 KB)

Whereas thumbor very much was not okay (5xx compared to loads from swift)

2026-08-13-123249_3640x1460_scrot.png (3,640×1,460 px, 211 KB)

2026-08-13-123259_3700x1241_scrot.png (3,700×1,241 px, 223 KB)

Change #1325528 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/alerts@master] sre/cdn: use traffic ratios in ATSBackendErrorsHigh

https://gerrit.wikimedia.org/r/1325528