Page MenuHomePhabricator

Send ORES logs to logstash
Closed, ResolvedPublic

Event Timeline

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptOct 24 2016, 8:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 321096 had a related patch set uploaded (by Ladsgroup):
ores: Send logs to logstash

https://gerrit.wikimedia.org/r/321096

Change 321096 merged by Alexandros Kosiaris:
[operations/puppet] service: Send uwsgi logs to logstash

https://gerrit.wikimedia.org/r/321096

While the change has been merged, and logs still make it to the local disk I don't see anything in logstash yet

I think I know why it's happening. See: T143172#2577829

I theory the log-route settings are not needed, but in practice I couldn't get everything to work as desired without them.

I make a patch for that.

Ladsgroup moved this task from Done to Blocked on others on the User-Ladsgroup board.

Change 344381 had a related patch set uploaded (by Alexandros Kosiaris):
[operations/puppet@production] service::uwsgi: name the local logger

https://gerrit.wikimedia.org/r/344381

Change 344381 merged by Alexandros Kosiaris:
[operations/puppet@production] service::uwsgi: name the local logger

https://gerrit.wikimedia.org/r/344381

Actually it was not that. I had to remove the log-route directives to make logging (neither logstash nor local were working) work again in the first place (sorry for not mentioning that earlier). That was in rOPUPb26323694f5e8, after which local logging worked out (but not logstash). I 've had a better look and finally spotted the typo. It was an omission for the local logger name. Fixed in rOPUPdf28e246647d4e . I 've reverted my log-route removal in rOPUP1131a7f660901.

We now have both local logging and logs are sent to logstash (verified by tcpdump).

That being said, I don't yet see however logs in logstash, but I do get this in the logstash logs

{:timestamp=>"2017-03-23T14:06:21.291000+0000", :message=>"Gelfd failed to parse a message skipping", :exception=>#<Gelfd::UnknownHeaderError: Could not find parser for header: [123, 34]>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/gelfd-0.2.0/lib/gelfd/parser.rb:14:in `parse'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-1.0.0/lib/logstash/inputs/gelf.rb:100:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-1.0.0/lib/logstash/inputs/gelf.rb:62:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/pipeline.rb:177:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/pipeline.rb:171:in `start_input'"], :level=>:warn}

Change 343772 abandoned by Alexandros Kosiaris:
service: add log-route in uwsgi config

Reason:
Fixed by a revert in https://gerrit.wikimedia.org/r/#/c/344378/

https://gerrit.wikimedia.org/r/343772

Change 344407 had a related patch set uploaded (by Alexandros Kosiaris):
[operations/puppet@production] logstash: Filter ores logstash messages and set port

https://gerrit.wikimedia.org/r/344407

Halfak removed Ladsgroup as the assignee of this task.Mar 27 2017, 4:58 PM

Change 344407 merged by Alexandros Kosiaris:
[operations/puppet@production] logstash: Filter ORES logstash messages and set port

https://gerrit.wikimedia.org/r/344407

From logstahs: [pid: 28235] 10.2.2.10 (-) {32 vars in 699 bytes} [Tue Mar 28 08:50:44 2017] GET /v2/scores/enwiki/?models=reverted%7Cdamaging%7Cgoodfaith&revids=772608446&precache=true&format=json => generated 495 bytes in 2140 msecs (HTTP/1.1 200) 6 headers in 208 bytes (1 switches on core 0) user agent "ChangePropagation/WMF"

\o/

Ladsgroup closed this task as Resolved.Mar 28 2017, 8:53 AM
Ladsgroup assigned this task to akosiaris.
Ladsgroup triaged this task as Normal priority.

Heh, you beat me to it :-)

I 've also created https://logstash.wikimedia.org/app/kibana#/dashboard/ORES and added a link from the home page.

I am wondering however how we could change the logformat to support more level (like ERROR, WARN) etc.