I'd be happy to write the patch, if an Ops engineer wants to suggest the preferred way to give us access?

The preferred way would be if logs can be read using systemd's journalctl and in the admin module in the section for ores-admin, a line like this is added to sudo privileges:

'ALL = NOPASSWD: /bin/journalctl *'

Another way would be a specific sudo privileges line to tail -f /srv/log/ores/main.log specifically

I would suggest AGAINST giving access to all logs. We should have a tail-ores command that specifically tails the ores logs, like we do with other services.

@Joe Great, thanks for the pointer! I can put together that tail wrapper shortly, but in the meantime I'm working on T169586 which will give us access to all the non-debug logs via logstash.

Change 377794 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] chmod uwsgi logs to 0644

https://gerrit.wikimedia.org/r/377794

The above gerrit change should make the main.log files readable by everyone and should fix the issue while aligning with the rest of the files for the various applications without the need for weird tricks. Note that these logs also make it logstash at all levels so it should be easy to view them from there too. The celery logs are not yet though

@akosiaris Could you point me to the config that propagates those log messages to logstash? I was looking at error messages from yesterday that didn't appear in logstash, see T175653 for details.

That would be https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/service/manifests/uwsgi.pp;c5f42c1ab6cbe51529637b5830c6d20294f57057$139

Since there are variables there here's the resulting uwsgi config on a host for your convenience

log-encoder=format:local [${strftime:%%Y-%%m-%%dT%%H:%%M:%%S}] ${msgnl}
log-encoder=json:logstash {"@timestamp":"${strftime:%%Y-%%m-%%dT%%H:%%M:%%S}","type":"ores","logger_name":"uwsgi","host":"%h","level":"INFO","message":"${msg}"}
log-route=local .*
log-route=logstash .*
logformat=[pid: %(pid)] %(addr) (%(user)) {%(vars) vars in %(pktsize) bytes} [%(ctime)] %(method) %(uri) => generated %(rsize) bytes in %(msecs) msecs (%(proto) %(status)) %(headers) headers in %(hsize) bytes (%(switches) switches on core %(core)) user agent "%(uagent)"
logger=local file:/srv/log/ores/main.log
logger=logstash socket:logstash1001.eqiad.wmnet:11514

@akosiaris Thanks, I see the messages in logstash now! I'll tweak the ORES dashboard to include them.

Correction to my last comment—the logs are already in the ORES dashboard of logstash.

Change 377794 abandoned by Alexandros Kosiaris:
chmod uwsgi logs to 0644

Reason:
Funnily enough, this won't work. logfile-chmod, logfile-chown have nothing to do with the logfile plugin but rather with the --log parameters. Abandoning

https://gerrit.wikimedia.org/r/377794