For the record, our current SLO is 95% uptime, which allows for 4.5 days of downtime per quarter, reflecting that the service can be down for a full weekend on occasion.

Change #1192940 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/alerts@master] team-sre: cdn: add wdqs-main.discovery.wmnet to ignored backends

https://gerrit.wikimedia.org/r/1192940

A few notes:

• wdqs-scholarly.discovery.wmnet and wdqs-main.discovery.wmnet are the 2 WDQS public endpoints. Both of those have a 95% uptime SLO, and should not page anyone.
• https://query-legacy-full.wikidata.org is a public endpoint to help scholia during the transition to a split graph. It is considered experimental, with no SLO. It is served by a single server and does not go through LVS.
• wcqs.discovery.wmnet is a public endpoint similar to WDQS, but serving data from Commons. It is authenticated and sees mostly no traffic, so it tends to not crash. It is considered experimental, with no SLO. It should not page anyone.

I'm not entirely sure which alerts paged during the last WDQS outage. I think it was about the number of servers depooled from LVS and not from error rate. The currently linked gerrit patch removes pages in cases of high number of backend errors, but I don't think it removes the page in case of too many servers depooled.

In T406141#11235857, @Gehel wrote:

I'm not entirely sure which alerts paged during the last WDQS outage. I think it was about the number of servers depooled from LVS and not from error rate. The currently linked gerrit patch removes pages in cases of high number of backend errors, but I don't think it removes the page in case of too many servers depooled.

At least in the most recent paging, it was wdqs-main.discovery.wmnet.

Thanks for the mention of the other services; I have updated the CR per your comment.

Now that the other endpoints were added, is there anything else that needs to happen before the patch is deployed?

In T406141#11249404, @LSobanski wrote:

Now that the other endpoints were added, is there anything else that needs to happen before the patch is deployed?

I am waiting for a review there and will ping @Gehel and @bking on the CR itself.

Change #1192940 merged by Ssingh:

[operations/alerts@master] team-sre/cdn: ignore (wdqs-main|wdqs-scholarly|wcqs).discovery.wmnet in ATSBackendErrorsHigh

https://gerrit.wikimedia.org/r/1192940

Thanks to a review by @bking, we have merged this. I think we can consider this as resolved. Thanks @LSobanski and @Gehel.