Around 11:45 UTC on 2016-12-18 three Cassandra instances OOMed: restbase1009-a, restbasr1011-c and restbase1013-c. This caused RESTBase to start returning 5xx responses for a portion of the requests (local quorum couldn't be achieved). We need to investigate what is the reason for the OOMs.
$ for i in 1009 1013 1011; do echo "$i: "; ssh restbase$i.eqiad.wmnet -- "sudo find /srv/cassandra-* -maxdepth 1 -name '*.hprof' -exec ls -lh {} \;"; done 1009: -rw------- 1 cassandra cassandra 9.4G Dec 18 16:52 /srv/cassandra-a/java_pid93714.hprof -rw------- 1 cassandra cassandra 10G Dec 18 11:19 /srv/cassandra-a/java_pid132816.hprof -rw------- 1 cassandra cassandra 9.1G Dec 18 08:57 /srv/cassandra-a/java_pid67908.hprof -rw------- 1 cassandra cassandra 11G Dec 18 10:03 /srv/cassandra-a/java_pid79638.hprof -rw------- 1 cassandra cassandra 8.8G Dec 18 10:59 /srv/cassandra-a/java_pid119356.hprof -rw------- 1 cassandra cassandra 11G Dec 18 15:35 /srv/cassandra-a/java_pid17899.hprof -rw------- 1 cassandra cassandra 11G Dec 18 10:19 /srv/cassandra-a/java_pid106595.hprof -rw------- 1 cassandra cassandra 11G Dec 18 12:07 /srv/cassandra-a/java_pid146258.hprof -rw------- 1 cassandra cassandra 11G Dec 18 08:24 /srv/cassandra-a/java_pid50562.hprof -rw------- 1 cassandra cassandra 11G Dec 18 08:44 /srv/cassandra-a/java_pid56179.hprof 1013: -rw------- 1 cassandra cassandra 11G Dec 18 12:08 /srv/cassandra-c/java_pid1824.hprof -rw------- 1 cassandra cassandra 9.9G Dec 18 10:11 /srv/cassandra-c/java_pid2732.hprof -rw------- 1 cassandra cassandra 11G Dec 18 10:04 /srv/cassandra-c/java_pid6892.hprof -rw------- 1 cassandra cassandra 7.9G Dec 18 01:29 /srv/cassandra-c/java_pid13269.hprof -rw------- 1 cassandra cassandra 11G Dec 18 10:59 /srv/cassandra-c/java_pid18131.hprof -rw------- 1 cassandra cassandra 11G Dec 18 08:38 /srv/cassandra-c/java_pid17470.hprof -rw------- 1 cassandra cassandra 11G Dec 18 11:44 /srv/cassandra-c/java_pid18013.hprof -rw------- 1 cassandra cassandra 8.7G Dec 18 08:58 /srv/cassandra-c/java_pid29353.hprof -rw------- 1 cassandra cassandra 11G Dec 18 08:25 /srv/cassandra-c/java_pid7954.hprof -rw------- 1 cassandra cassandra 11G Dec 18 11:20 /srv/cassandra-c/java_pid1880.hprof 1011: -rw------- 1 cassandra cassandra 9.9G Dec 18 15:35 /srv/cassandra-c/java_pid3363.hprof -rw------- 1 cassandra cassandra 7.3G Dec 18 01:28 /srv/cassandra-c/java_pid6380.hprof -rw------- 1 cassandra cassandra 11G Dec 18 11:43 /srv/cassandra-c/java_pid23170.hprof -rw------- 1 cassandra cassandra 11G Dec 18 16:58 /srv/cassandra-c/java_pid6608.hprof -rw------- 1 cassandra cassandra 11G Dec 18 08:38 /srv/cassandra-c/java_pid21659.hprof -rw------- 1 cassandra cassandra 11G Dec 18 10:19 /srv/cassandra-c/java_pid7690.hprof -rw------- 1 cassandra cassandra 8.4G Dec 18 08:44 /srv/cassandra-c/java_pid30320.hprof -rw------- 1 cassandra cassandra 9.1G Dec 18 16:52 /srv/cassandra-c/java_pid1220.hprof