With the current load we are seeing multiple Java OOM failures for Elasticsearch per day and the boxes are so busy doing GC runs that they aren't keeping up with the inbound log traffic.
Each host is currently 16G; taking that up to 64G will help greatly.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | bd808 | T69817 Monitor for anomalies/spikes in read failures of memcached | |||
| Resolved | bd808 | T100735 Have Logstash report per-channel log message rate to Graphite | |||
| Resolved | bd808 | T99735 Upgrade Logstash to 1.5.3 | |||
| Resolved | bd808 | T97545 reinstall logstash1001-1003 | |||
| Resolved | Anomie | T1272 Deploy ApiFeatureUsage extension on WMF wikis | |||
| Resolved | bd808 | T87521 Convert JobRunner.php to PSR-3 logging and add levels | |||
| Resolved | bd808 | T88732 Decouple logging infrastructure failures from MediaWiki logging | |||
| Resolved | bd808 | T96692 Rack and Setup (3) Logstash Servers | |||
| Resolved | RobH | T84958 eqiad: (3) servers for logstash service | |||
| Declined | bd808 | T87078 Upgrade RAM for logstash100[123] to 64G | |||
| Resolved | faidon | T97481 jessie installs fail - mirror issue due to jessie release? | |||
| Resolved | bd808 | T97645 Elasticsearch not starting on Jessie hosts |
we rejected this since there are 6 slots open for memory, and 4 use the 4GB. we'd be replacing ALL system memory for 3 hosts which end their warranty support this fall 2015.