Page MenuHomePhabricator
Create Task
Maniphest T76090

Icinga monitoring for elasticsearch doesn't notice OOM conditions (this is happening on cloud)
Closed, ResolvedPublic
Actions

Description

The monitoring for Elasticsearch doesn't seem to notice when an OOM has happened on a node. I have very seldom seen a node recover on it's own from an OOM so it seems like something worth of alerting about.

Details

SubjectRepoBranchLines +/-
elasticsearch: exit the JVM on OutOfMemoryErroroperations/puppetproduction+3 -0
Customize query in gerrit

Related Objects

Event Timeline

bd808 created this task.Nov 26 2014, 9:13 PM
bd808 raised the priority of this task from to Needs Triage.
bd808 updated the task description. (Show Details)
bd808 added a project: Wikimedia-Logstash.
bd808 changed Security from none to None.
bd808 added a project: Release-Engineering-Team.
bd808 added subscribers: bd808, • Gage, • Manybubbles.
bd808 added a comment.Nov 26 2014, 9:17 PM

I think I remember there being a java jre command line flag that lets you install an OOM signal handler. I have vague memories of using that to send alerts in a long forgotten past as a Java Shop Administrator™.

• Gage added a project: observability.Dec 18 2014, 8:00 PM
bd808 triaged this task as Medium priority.Feb 2 2015, 5:07 PM
hashar edited projects, added acl*sre-team, Elasticsearch; removed Release-Engineering-Team.May 29 2015, 10:55 AM
hashar subscribed.

Moving that monitoring task from Release-Engineering-Team to SRE

• chasemp mentioned this in T109089: EPIC: Cultivating the Elasticsearch garden (operational lessons from 1.7.1 upgrade).Aug 14 2015, 6:31 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 7:50 PM
Restricted Application added a project: Discovery-Search. · View Herald TranscriptJan 26 2019, 7:50 PM
gerritbot subscribed.Feb 3 2019, 11:25 PM

Change 487787 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: exit the JVM on OutOfMemoryError

https://gerrit.wikimedia.org/r/487787

gerritbot added a project: Patch-For-Review.Feb 3 2019, 11:25 PM
Gehel subscribed.Feb 3 2019, 11:26 PM

Instead of monitoring this specific error, let's just configure the JVM to restart on memory errors.

Gehel claimed this task.Feb 3 2019, 11:26 PM
Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.
Gehel moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.
hashar unsubscribed.Feb 4 2019, 11:58 AM
• Nuria renamed this task from Icinga monitoring for elasticsearch doesn't notice OOM conditions to Icinga monitoring for elasticsearch doesn't notice OOM conditions (this is happening on cloud) .Mar 5 2019, 6:28 PM
gerritbot added a comment.Mar 6 2019, 9:49 AM

Change 487787 merged by Gehel:
[operations/puppet@production] elasticsearch: exit the JVM on OutOfMemoryError

https://gerrit.wikimedia.org/r/487787

Gehel moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Mar 6 2019, 10:10 AM

Merged, will take effect with the next cluster restarts.

debt closed this task as Resolved.Mar 8 2019, 5:52 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 19 2019, 3:17 PM
hashar mentioned this in T267517: gerrit-replica was down for a while and no one noticed.Nov 9 2020, 9:25 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL