Page MenuHomePhabricator
Create Task
Maniphest T389859

WDQS: Alert on high thread count
Open, In Progress, MediumPublic
Actions

Description

The full graph WDQS hosts in eqiad appear to have suffered a cascading failure starting at about 1900 UTC on 23 March 2025 ( ref this graph ; when a specific host's lag metrics disappear from the graph, that means it stopped working).

This particular failure scenario did not trigger any alerts until the entire service was lagging, at which point we got ElevatedMaxLagWDQS: WDQS lag is above 10 minutes alerts. Creating this ticket to:

  • Decide on the best alert or alerts for this failure scenario

^^ We'll try a thread count alert

  • Implement the alerts and verify operation

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wdqs: detect blazegraph deadlockoperations/alertsmaster+49 -0
WIP: wdqs: Add alerts for no lag metrics reportedoperations/alertsmaster+33 -0
Customize query in gerrit

Related Objects

Event Timeline

bking created this task.Mar 24 2025, 6:52 PM
bking triaged this task as Medium priority.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 24 2025, 6:52 PM
bking renamed this task from WDQS: Alert on high thread count to WDQS: Alert on high thread count or no lag metrics reported.Mar 24 2025, 7:06 PM
bking updated the task description. (Show Details)
bking updated the task description. (Show Details)
bking updated the task description. (Show Details)Mar 24 2025, 7:34 PM
bking added a subscriber: dcausse.
gerritbot added a comment.Mar 24 2025, 10:31 PM

Change #1130730 had a related patch set uploaded (by Bking; author: Bking):

[operations/alerts@master] WIP: wdqs: Add alerts for no lag metrics reported

https://gerrit.wikimedia.org/r/1130730

gerritbot added a project: Patch-For-Review.Mar 24 2025, 10:31 PM
bking changed the task status from Open to In Progress.Mar 25 2025, 1:15 PM
bking claimed this task.
Gehel edited projects, added Data-Platform-SRE (2025.03.22 - 2025.04.11); removed Data-Platform-SRE.Mar 25 2025, 1:53 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.03.22 - 2025.04.11) board.
Gehel subscribed.Apr 7 2025, 3:18 PM

Related: T270614

bking added a comment.Apr 7 2025, 3:27 PM

Some notes from our discussion at today's Search Platform standup:

  • We got ProbeDown alerts for these hosts Sunday night, SRE needs to be more responsive on Monday morning
  • We could have an alert on thread count.
  • We've also talked about automatically depooling lagged hosts (see T270614 ). This ticket is already on the Data Platform SRE workboard.
bking updated the task description. (Show Details)Apr 7 2025, 3:28 PM
Gehel edited projects, added Data-Platform-SRE (2025.04.12 - 2025.05.02); removed Data-Platform-SRE (2025.03.22 - 2025.04.11).Apr 11 2025, 1:14 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.04.12 - 2025.05.02) board.
bking reassigned this task from bking to RKemper.Apr 15 2025, 1:52 PM
bking added a subscriber: RKemper.

@RKemper Per 1x1 with @Gehel today, I haven't done a good job of moving this forward. We were wondering if we could delegate this one to you? I'm putting it in your name, but feel free to ping back if that doesn't work for you.

Gehel edited projects, added Data-Platform-SRE (2025.05.02 - 2025.05.23); removed Data-Platform-SRE (2025.04.12 - 2025.05.02).May 5 2025, 12:49 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.
Gehel edited projects, added Data-Platform-SRE (2025.05.24 - 2025.06.13); removed Data-Platform-SRE (2025.05.02 - 2025.05.23).May 23 2025, 1:08 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.05.24 - 2025.06.13) board.
Gehel added a comment.Jun 5 2025, 6:42 PM

Thread count graph: https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&from=now-7d&to=now&timezone=utc&var-cluster_name=wdqs-main&var-graph_type=%289102%7C919%5B35%5D%29&refresh=1m&viewPanel=panel-22

Gehel edited projects, added Data-Platform-SRE (2025.06.13 - 2025.07.04); removed Data-Platform-SRE (2025.05.24 - 2025.06.13).Jun 13 2025, 8:49 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.
BTullis edited projects, added Data-Platform-SRE (2025.07.05 - 2025.07.25); removed Data-Platform-SRE (2025.06.13 - 2025.07.04).Jul 4 2025, 3:59 PM
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 4 2025, 4:20 PM
BTullis edited projects, added Data-Platform-SRE (2025.07.26 - 2025.08.15); removed Data-Platform-SRE (2025.07.05 - 2025.07.25).Jul 28 2025, 4:47 PM
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.
BTullis edited projects, added Data-Platform-SRE (2025.08.16 - 2025.09.05); removed Data-Platform-SRE (2025.07.26 - 2025.08.15).Aug 20 2025, 10:30 AM
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.08.16 - 2025.09.05) board.
Gehel edited projects, added Data-Platform-SRE (2025.09.05 - 2025.09.26); removed Data-Platform-SRE (2025.08.16 - 2025.09.05).Sep 5 2025, 2:24 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.
gerritbot added a comment.Sep 10 2025, 5:41 PM

Change #1130730 abandoned by Bking:

[operations/alerts@master] WIP: wdqs: Add alerts for no lag metrics reported

Reason:

won't work and probably not needed

https://gerrit.wikimedia.org/r/1130730

bking renamed this task from WDQS: Alert on high thread count or no lag metrics reported to WDQS: Alert on high thread count.Sep 10 2025, 5:42 PM
Maintenance_bot removed a project: Patch-For-Review.Sep 10 2025, 6:32 PM
Gehel added a project: Essential-Work.Sep 11 2025, 12:04 PM
Gehel edited projects, added Data-Platform-SRE (2025.09.26 - 2025.10.17); removed Data-Platform-SRE (2025.09.05 - 2025.09.26).Sep 26 2025, 1:51 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.
Gehel edited projects, added Data-Platform-SRE (2025.10.17 - 2025.11.07); removed Data-Platform-SRE (2025.09.26 - 2025.10.17).Fri, Oct 17, 8:49 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.10.17 - 2025.11.07) board.
gerritbot added a comment.Wed, Oct 22, 10:05 PM

Change #1198161 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/alerts@master] (wip) wdqs: detect blazegraph deadlock

https://gerrit.wikimedia.org/r/1198161

gerritbot added a project: Patch-For-Review.Wed, Oct 22, 10:05 PM
RKemper added a comment.Thu, Oct 23, 7:13 AM

Uploaded a proposed solution that alerts if the triples count metric is missing. Blazegraph deadlock always leads to the triple count metric missing.

I'm not sure if this current proposed solution is robust to firing if a previously-extant host is decommissioned. We might need to make the alert more complex to account for this possibility.

RKemper added a comment.Thu, Oct 23, 8:28 AM

With respect to the thread-count approach, here's an interesting graph of a recent deadlock: https://grafana-rw.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&refresh=1m&var-cluster_name=wdqs-main&from=2025-10-22T20:47:05.347Z&to=2025-10-23T08:26:43.628Z&timezone=utc&var-graph_type=%289102%7C919%5B35%5D%29&viewPanel=panel-22

RKemper added a comment.Mon, Nov 3, 10:40 PM

https://grafana-rw.wikimedia.org/explore?schemaVersion=1&panes=%7B%22ry8%22%3A%7B%22datasource%22%3A%22000000026%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22avg_over_time%28jvm_threads_current%7Bcluster%3D%5C%22wdqs-main%5C%22%2Cjob%3D%7E%5C%22jmx_w%5Bcd%5Dqs_blazegraph%5C%22%7D%5B5m%5D%29%22%2C%22range%22%3Atrue%2C%22instant%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22000000026%22%7D%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22__auto%22%7D%5D%2C%22range%22%3A%7B%22from%22%3A%221761166025347%22%2C%22to%22%3A%221761208003628%22%7D%7D%7D&orgId=1

We could alert when this metric hits >1000 or maybe >1500

Gehel edited projects, added Data-Platform-SRE (2025.11.07 - 2025.11.28); removed Data-Platform-SRE (2025.10.17 - 2025.11.07).Fri, Nov 7, 1:31 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.11.07 - 2025.11.28) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits