Page MenuHomePhabricator

Upgrade RAM for logstash100[123] to 64G
Closed, DeclinedPublic

Description

With the current load we are seeing multiple Java OOM failures for Elasticsearch per day and the boxes are so busy doing GC runs that they aren't keeping up with the inbound log traffic.

http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Logstash%20cluster%20eqiad&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1421455397&g=mem_report&z=large

Each host is currently 16G; taking that up to 64G will help greatly.

Event Timeline

bd808 created this task.Jan 17 2015, 12:45 AM
bd808 raised the priority of this task from to Normal.
bd808 updated the task description. (Show Details)
bd808 added subscribers: Gage, bd808, MZMcBride and 4 others.
ori updated the task description. (Show Details)Jan 17 2015, 12:57 AM
ori set Security to None.
bd808 moved this task from Backlog to Archive on the Wikimedia-Logstash board.Apr 22 2015, 3:25 AM
RobH added a subscriber: RobH.Apr 29 2015, 3:52 PM

we rejected this since there are 6 slots open for memory, and 4 use the 4GB. we'd be replacing ALL system memory for 3 hosts which end their warranty support this fall 2015.

Restricted Application added a project: User-bd808. · View Herald TranscriptJul 29 2016, 8:19 PM
bd808 moved this task from To Do to Archive on the User-bd808 board.Sep 1 2016, 5:15 PM