Description
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | Feature | None | T63779 Add system logs to logstash (tracking) | ||
| Resolved | akosiaris | T149010 Send ORES logs to logstash |
Event Timeline
Change 321096 had a related patch set uploaded (by Ladsgroup):
ores: Send logs to logstash
Change 321096 merged by Alexandros Kosiaris:
[operations/puppet] service: Send uwsgi logs to logstash
While the change has been merged, and logs still make it to the local disk I don't see anything in logstash yet
I think I know why it's happening. See: T143172#2577829
I theory the log-route settings are not needed, but in practice I couldn't get everything to work as desired without them.
I make a patch for that.
Change 344381 had a related patch set uploaded (by Alexandros Kosiaris):
[operations/puppet@production] service::uwsgi: name the local logger
Change 344381 merged by Alexandros Kosiaris:
[operations/puppet@production] service::uwsgi: name the local logger
Actually it was not that. I had to remove the log-route directives to make logging (neither logstash nor local were working) work again in the first place (sorry for not mentioning that earlier). That was in rOPUPb26323694f5e8, after which local logging worked out (but not logstash). I 've had a better look and finally spotted the typo. It was an omission for the local logger name. Fixed in rOPUPdf28e246647d4e . I 've reverted my log-route removal in rOPUP1131a7f660901.
We now have both local logging and logs are sent to logstash (verified by tcpdump).
That being said, I don't yet see however logs in logstash, but I do get this in the logstash logs
{:timestamp=>"2017-03-23T14:06:21.291000+0000", :message=>"Gelfd failed to parse a message skipping", :exception=>#<Gelfd::UnknownHeaderError: Could not find parser for header: [123, 34]>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/gelfd-0.2.0/lib/gelfd/parser.rb:14:in `parse'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-1.0.0/lib/logstash/inputs/gelf.rb:100:in `udp_listener'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-gelf-1.0.0/lib/logstash/inputs/gelf.rb:62:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/pipeline.rb:177:in `inputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/pipeline.rb:171:in `start_input'"], :level=>:warn}Change 343772 abandoned by Alexandros Kosiaris:
service: add log-route in uwsgi config
Reason:
Fixed by a revert in https://gerrit.wikimedia.org/r/#/c/344378/
Change 344407 had a related patch set uploaded (by Alexandros Kosiaris):
[operations/puppet@production] logstash: Filter ores logstash messages and set port
Change 344407 merged by Alexandros Kosiaris:
[operations/puppet@production] logstash: Filter ORES logstash messages and set port
From logstahs: [pid: 28235] 10.2.2.10 (-) {32 vars in 699 bytes} [Tue Mar 28 08:50:44 2017] GET /v2/scores/enwiki/?models=reverted%7Cdamaging%7Cgoodfaith&revids=772608446&precache=true&format=json => generated 495 bytes in 2140 msecs (HTTP/1.1 200) 6 headers in 208 bytes (1 switches on core 0) user agent "ChangePropagation/WMF"
\o/
Heh, you beat me to it :-)
I 've also created https://logstash.wikimedia.org/app/kibana#/dashboard/ORES and added a link from the home page.
I am wondering however how we could change the logformat to support more level (like ERROR, WARN) etc.