Page MenuHomePhabricator
Create Task
Maniphest T387745

Improve monitoring/update dashboards/update docs for Elastic
Open, In Progress, Needs TriagePublic
Actions

Description

While troubleshooting Elastic cluster latency in the parent task ( T387176 ), we've noticed a few things that could speed up diagnoses in the future:

Creating this ticket to fulfill the above AC.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
cirrussearch: Add alerts for thread pool rejectionsBkingoperations/alertsmaster+39 -0
Customize query in gerrit

Related Objects

Event Timeline

bking created this task.Mar 3 2025, 3:43 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 3 2025, 3:43 PM
Gehel moved this task from needs triage to Ops / SRE on the Discovery-Search board.Mar 3 2025, 4:29 PM
bking added a comment.Mar 5 2025, 10:32 PM

We can use the following prometheus query for alerting on per-node thread pool rejections:

elasticsearch_thread_pool_rejected_count{instance="elastic1066:9108"}

gerritbot added a comment.Mar 6 2025, 3:49 PM

Change #1125180 had a related patch set uploaded (by Bking; author: Bking):

[operations/alerts@master] cirrussearch: Add alerts for thread pool rejections

https://gerrit.wikimedia.org/r/1125180

gerritbot added a project: Patch-For-Review.Mar 6 2025, 3:49 PM
bking updated the task description. (Show Details)Mar 6 2025, 8:05 PM
gerritbot added a comment.Mar 17 2025, 3:49 PM

Change #1125180 merged by jenkins-bot:

[operations/alerts@master] cirrussearch: Add alerts for thread pool rejections

https://gerrit.wikimedia.org/r/1125180

bking updated the task description. (Show Details)Mar 17 2025, 3:59 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 17 2025, 4:30 PM
bking changed the task status from Open to In Progress.Mar 17 2025, 5:27 PM
bking claimed this task.
Gehel edited projects, added: Data-Platform-SRE (2025.03.22 - 2025.04.11); removed: Data-Platform-SRE (2025.03.01 - 2025.03.21).Mar 21 2025, 9:57 AM
Gehel edited projects, added: Data-Platform-SRE (2025.04.12 - 2025.05.02); removed: Data-Platform-SRE (2025.03.22 - 2025.04.11).Apr 11 2025, 1:14 PM
Gehel edited projects, added: Data-Platform-SRE (2025.05.02 - 2025.05.23); removed: Data-Platform-SRE (2025.04.12 - 2025.05.02).May 5 2025, 12:49 PM
Gehel removed bking as the assignee of this task.May 7 2025, 1:46 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.05.02 - 2025.05.23) board.
Gehel edited projects, added: Data-Platform-SRE (2025.05.24 - 2025.06.13); removed: Data-Platform-SRE (2025.05.02 - 2025.05.23).May 23 2025, 1:08 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.05.24 - 2025.06.13) board.
Gehel edited projects, added: Data-Platform-SRE (2025.06.13 - 2025.07.04); removed: Data-Platform-SRE (2025.05.24 - 2025.06.13).Jun 13 2025, 8:49 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.06.13 - 2025.07.04) board.
BTullis edited projects, added: Data-Platform-SRE (2025.07.05 - 2025.07.25); removed: Data-Platform-SRE (2025.06.13 - 2025.07.04).Jul 4 2025, 3:59 PM
BTullis moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.07.05 - 2025.07.25) board.Jul 4 2025, 4:22 PM
BTullis edited projects, added: Data-Platform-SRE (2025.07.26 - 2025.08.15); removed: Data-Platform-SRE (2025.07.05 - 2025.07.25).Jul 28 2025, 4:48 PM
BTullis moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.
BTullis edited projects, added: Data-Platform-SRE (2025.08.16 - 2025.09.05); removed: Data-Platform-SRE (2025.07.26 - 2025.08.15).Aug 20 2025, 10:29 AM
BTullis moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.08.16 - 2025.09.05) board.
Gehel edited projects, added: Data-Platform-SRE (2025.09.05 - 2025.09.26); removed: Data-Platform-SRE (2025.08.16 - 2025.09.05).Sep 5 2025, 2:24 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.
Gehel added a project: Essential-Work.Sep 11 2025, 12:10 PM
Gehel edited projects, added: Data-Platform-SRE (2025.09.26 - 2025.10.17); removed: Data-Platform-SRE (2025.09.05 - 2025.09.26).Sep 26 2025, 1:51 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.
Gehel edited projects, added: Data-Platform-SRE (2025.10.17 - 2025.11.07); removed: Data-Platform-SRE (2025.09.26 - 2025.10.17).Oct 17 2025, 8:50 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.10.17 - 2025.11.07) board.Oct 17 2025, 8:52 AM
Gehel edited projects, added: Data-Platform-SRE (2025.11.07 - 2025.11.28); removed: Data-Platform-SRE (2025.10.17 - 2025.11.07).Nov 7 2025, 1:32 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.11.07 - 2025.11.28) board.
Gehel edited projects, added: Data-Platform-SRE (2026.01.05 - 2026.01.23); removed: Data-Platform-SRE (2025.11.07 - 2025.11.28).Jan 8 2026, 1:15 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026.01.05 - 2026.01.23) board.
Gehel edited projects, added: Data-Platform-SRE (2026.01.23 - 2026.02.13); removed: Data-Platform-SRE (2026.01.05 - 2026.01.23).Jan 23 2026, 10:04 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026.01.23 - 2026.02.13) board.
Gehel edited projects, added: Data-Platform-SRE (2026-02-13 - 2026-03-06); removed: Data-Platform-SRE (2026.01.23 - 2026.02.13).Feb 16 2026, 3:36 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-02-13 - 2026-03-06) board.
Gehel edited projects, added: Data-Platform-SRE (2026-03-06 - 2026-03-27); removed: Data-Platform-SRE (2026-02-13 - 2026-03-06).Mar 6 2026, 2:03 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-03-06 - 2026-03-27) board.
Gehel edited projects, added: Data-Platform-SRE (2026-03-27 - 2026-04-17); removed: Data-Platform-SRE (2026-03-06 - 2026-03-27).Mar 27 2026, 9:50 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-03-27 - 2026-04-17) board.
Gehel edited projects, added: Data-Platform-SRE (2026-04-24 - 2026-05-15); removed: Data-Platform-SRE (2026-03-27 - 2026-04-17).Apr 24 2026, 9:58 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.
Gehel edited projects, added: Data-Platform-SRE (2026-06-05 - 2026-06-26); removed: Data-Platform-SRE (2026-04-24 - 2026-05-15).Jun 5 2026, 1:25 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.
Gehel edited projects, added: Data-Platform-SRE (2026-07-03 - 2026-07-31); removed: Data-Platform-SRE (2026-06-05 - 2026-06-26).Jul 3 2026, 9:50 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-07-03 - 2026-07-31) board.
Gehel edited projects, added: Data-Platform-SRE (2026-08-07 - 2026-08-28); removed: Data-Platform-SRE (2026-07-03 - 2026-07-31).Tue, Aug 11, 6:52 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2026-08-07 - 2026-08-28) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits