Page MenuHomePhabricator

ORES log messages should be sent to logstash
Closed, DuplicatePublic


I'm seeing 100% 503 server overloaded errors from ores100*, but no error messages recorded in logstash. This could be a configuration or a source code bug.!f%2Cvalue%3A0)%2Ctime%3A(from%3A'2017-09-11T23%3A50%3A26.403Z'%2Cmode%3Aabsolute%2Cto%3A'2017-09-11T23%3A57%3A30.888Z'))

I see weblogs, but no application logs.

[pid: 14941] (-) {32 vars in 497 bytes} [Mon Sep 11 23:52:42 2017] GET /v2/scores/enwiki/damaging/12345678 => generated 160 bytes in 8 msecs (HTTP/1.1 503) 6 headers in 225 bytes (1 switches on core 0) user agent "curl/7.38.0"

According to the source code, when we see this particular 503 error (server overloaded), the following log message should go somewhere:

message = "Queue size is too full {0}".format(queue_size)

Our logging is configured in, e.g. ores1001.eqiad.wmnet:/srv/deployment/ores/deploy/logging_config.yaml . This should be easy to mirror to logstash. I'd prefer INFO-level and higher for now.

Event Timeline

awight renamed this task from ORES error messages not in logstash? to ORES log messages should be sent to logstash.Sep 12 2017, 7:29 PM
awight added a project: SRE.
awight updated the task description. (Show Details)

Change 377553 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/services/ores/deploy@master] [WIP] Send logs to logstash