While checking into the upgrade of logstash to 5.x i noticed a couple errors due to malformed GELF logging requests. This is explicitly *not* a problem with the 5.x upgrade, our 1.5.x install in production is logging the same errors, I just noticed these because i was looking over logs while preparing the upgrade.
These are basically udp messages formatted with json received over port 12201. One example message:
{"@timestamp":"2017-03-27T21:11:24","type":"ores","logger_name":"uwsgi","host":"deployment-sca03","level":"ERROR","message":"[pid: 31379] 10.68.21.68 (-) {32 vars in 521 bytes} [Mon Mar 27 21:11:24 2017] GET /scores/enwiki/goodfaith/?model_info=test_stats&format=json => generated 2060 bytes in 7 msecs (HTTP/1.1 200) 6 headers in 209 bytes (1 switches on core 0) user agent \"MediaWiki/1.29.0-alpha\""}The problem here is the logstash can only accept compressed input over GELF, plaintext is not supported. I'm no uwsgi expert so can't provide exact details on how to fix, but for the logs to be accepted by logstash and saved into elasticsearch, to be viewed in kibana, the uwsgi config in /etc/uwsgi/apps-available/ores.ini will need to be updated to compress the data sent out over the socket connection.