https://logstash.wikimedia.org/goto/2cd02481a31ab0903a3f057d9aa3fda6
Actual logs are produced and sent to fluorine with no issue:
fluorine:/a/mw-log$ egrep '^Jan 1 00:' -c hhvm.log 5886 fluorine:/a/mw-log$ egrep '^Jan 1 00:' -c hhvm.log 5898
Not sure yet if it is a production or a storage issue.
I have added more tags that this eventually should have, just as a heads up- as it is not clear where the issue is (client or server).
logstash1001:~ bd808$ sudo journalctl -l --no-pager -f -u logstash -- Logs begin at Thu 2016-12-22 07:12:58 UTC. -- Dec 24 23:25:33 logstash1001 logstash[32399]: Exception in thread "Ruby-0-Thread-18: /opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.20/lib/stud/buffer.rb:92" java.lang.UnsupportedOperationException Dec 24 23:25:33 logstash1001 logstash[32399]: at java.lang.Thread.stop(Thread.java:869) Dec 24 23:25:33 logstash1001 logstash[32399]: at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1221) Dec 24 23:25:33 logstash1001 logstash[32399]: at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:112) Dec 24 23:25:33 logstash1001 logstash[32399]: at java.lang.Thread.run(Thread.java:745)
Mentioned in SAL (#wikimedia-operations) [2017-01-01T00:55:19Z] <bd808> Restarted logstash on logstash1001 (T154388)