Change Details

With the current load we are seeing multiple Java OOM failures for Elasticsearch per day and the boxes are so busy doing GC runs that they aren't keeping up with the inbound log traffic. http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Logstash%20cluster%20eqiad&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1421455397&g=mem_report&z=large Each host is currently 16G; taking that up to 48/64/96G (whatever we have around and they will hold) will help greatly and should tide us over until a plan is made for replacement hardware.