were there any changes in logstash around that time?

SAL says there were (seemingly unrelated) changes wrt ganglia on the nodes on 2015-09-13, but that's it.

I didn't find anything obviously related in the puppet git log either.

Only candidates form SAL:

• some elasticsearch changes on the 15th
• lots of FERM firewalling, but mostly after the 14th as well

RESTBase is configured to log to logstash1001. (I don't think we have a service IP / LVS for logstash -- fixing that should be a priority.)

On logstash1001, I see the following in /var/log/logstash/logstash.err:

Error: Your application used more memory than the safety cap of 500M.
Specify -J-Xmx####m to increase it (#### = cap size in MB).
Specify -w for full OutOfMemoryError stack trace

The message is not timestamped, but the file was last modified on Sep 14 14:37.

The service appears to have exited then, and it has not been restarted:

$ service logstash status
● logstash.service - LSB: Starts Logstash as a daemon.
   Loaded: loaded (/etc/init.d/logstash)
   Active: active (exited) since Thu 2015-08-27 15:59:19 UTC; 3 weeks 0 days ago

Aug 27 15:59:19 logstash1001 logstash[629]: logstash started.
Aug 27 15:59:19 logstash1001 systemd[1]: Started LSB: Starts Logstash as a daemon..

logstash1001 is also the forward_syslog for MediaWiki, so I imagine MediaWiki logs are missing, too.

Change 239307 had a related patch set uploaded (by Ori.livneh):
logstash: add monitoring for logstash process

https://gerrit.wikimedia.org/r/239307

Change 239307 merged by Ori.livneh:
logstash: add monitoring for logstash process

https://gerrit.wikimedia.org/r/239307

Change 239506 had a related patch set uploaded (by GWicke):
Switch RESTBase logging to logstash1002.eqiad

https://gerrit.wikimedia.org/r/239506

Change 239506 merged by Ori.livneh:
Switch RESTBase logging to logstash1002.eqiad

https://gerrit.wikimedia.org/r/239506