No HHVM logs on kibana since 1 Jan 2017 0:00
Closed, ResolvedPublic

Description

https://logstash.wikimedia.org/goto/2cd02481a31ab0903a3f057d9aa3fda6

Actual logs are produced and sent to fluorine with no issue:

fluorine:/a/mw-log$ egrep '^Jan  1 00:' -c hhvm.log
5886
fluorine:/a/mw-log$ egrep '^Jan  1 00:' -c hhvm.log
5898

Not sure yet if it is a production or a storage issue.

jcrespo created this task.Jan 1 2017, 12:26 AM
Restricted Application added projects: Discovery, Discovery-Search. · View Herald TranscriptJan 1 2017, 12:26 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Gehel added a subscriber: Gehel.Jan 1 2017, 12:31 AM

I have added more tags that this eventually should have, just as a heads up- as it is not clear where the issue is (client or server).

bd808 added a subscriber: bd808.Jan 1 2017, 12:54 AM
logstash1001:~
bd808$ sudo journalctl -l --no-pager -f -u logstash
-- Logs begin at Thu 2016-12-22 07:12:58 UTC. --
Dec 24 23:25:33 logstash1001 logstash[32399]: Exception in thread "Ruby-0-Thread-18: /opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.20/lib/stud/buffer.rb:92" java.lang.UnsupportedOperationException
Dec 24 23:25:33 logstash1001 logstash[32399]: at java.lang.Thread.stop(Thread.java:869)
Dec 24 23:25:33 logstash1001 logstash[32399]: at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1221)
Dec 24 23:25:33 logstash1001 logstash[32399]: at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:112)
Dec 24 23:25:33 logstash1001 logstash[32399]: at java.lang.Thread.run(Thread.java:745)

Mentioned in SAL (#wikimedia-operations) [2017-01-01T00:55:19Z] <bd808> Restarted logstash on logstash1001 (T154388)

jcrespo closed this task as "Resolved".Jan 1 2017, 1:00 AM
jcrespo claimed this task.
jcrespo reassigned this task from jcrespo to bd808.
Restricted Application added a project: User-bd808. · View Herald TranscriptJan 1 2017, 1:02 AM
bd808 moved this task from To Do to Done on the User-bd808 board.Jan 7 2017, 12:18 AM