I'm seeing 100% 503 server overloaded errors from ores100*, but no error messages recorded in logstash. This could be a configuration or a source code bug.

https://logstash.wikimedia.org/app/kibana#/dashboard/ORES?_g=(refreshInterval%3A(display%3AOff%2Cpause%3A!f%2Cvalue%3A0)%2Ctime%3A(from%3A'2017-09-11T23%3A50%3A26.403Z'%2Cmode%3Aabsolute%2Cto%3A'2017-09-11T23%3A57%3A30.888Z'))

I see weblogs, but no application logs.

[pid: 14941] 208.80.154.22 (-) {32 vars in 497 bytes} [Mon Sep 11 23:52:42 2017] GET /v2/scores/enwiki/damaging/12345678 => generated 160 bytes in 8 msecs (HTTP/1.1 503) 6 headers in 225 bytes (1 switches on core 0) user agent "curl/7.38.0"

According to the source code, when we see this particular 503 error (server overloaded), the following log message should go somewhere:

message = "Queue size is too full {0}".format(queue_size)
logger.warning(message)

Our logging is configured in, e.g. ores1001.eqiad.wmnet:/srv/deployment/ores/deploy/logging_config.yaml . This should be easy to mirror to logstash. I'd prefer INFO-level and higher for now.