@fgiunchedi what's the current best practice here? Debmonitor is just using service::uwsgi that automatically logs to /srv/log/debmonitor/main.log and AFAIK doesn't log to the local syslog normal connections but just restarts of the daemon.

I guess we would like to use a shared solution for all the uwsgi servers we have.

In T245512#5892948, @Volans wrote:

@fgiunchedi what's the current best practice here? Debmonitor is just using service::uwsgi that automatically logs to /srv/log/debmonitor/main.log and AFAIK doesn't log to the local syslog normal connections but just restarts of the daemon.

Essentially two options: json to syslog/journald (works best on Buster, before that journald has max 2k line length limitation) or json to local udp syslog (via rsyslog and '::profile::rsyslog::udp_json_logback_compat'). For a standard solution that applies to all uwsgi servers I'm thinking probably the latter is the easiest ATM (alternatively wait for Buster migration), what do you think ?

I suggest that we implement the logic in service::uwsgi allowing for an easy opt-in from all the users of that define, that I bet is used on WMCS too.
For the implementation between the option if we have time probably nicer to implement directly the buster solution and apply it while migrating to it.
But if we want/need to push for a quicker migration then let's do the interim local udp that is compatible with stretch too.

Agreed, I'll send out patches to add switches to service::uwsgi for logging pipeline. I agree the optimal would be journald although I suspect the full Buster migration isn't imminent at this point, will try with rsyslog first.

Change 573936 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] service: fix uwsgi logstash_port_logback

https://gerrit.wikimedia.org/r/573936

Change 573937 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] service: logging pipeline support for uwsgi

https://gerrit.wikimedia.org/r/573937

Change 573936 merged by Filippo Giunchedi:
[operations/puppet@production] service: fix uwsgi logstash_port_logback

https://gerrit.wikimedia.org/r/573936

Change 574765 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] debmonitor: log to logging pipeline

https://gerrit.wikimedia.org/r/574765

Change 574766 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] puppetboard: log to logging pipeline

https://gerrit.wikimedia.org/r/574766

Change 574765 merged by Filippo Giunchedi:
[operations/puppet@production] debmonitor: log to logging pipeline

https://gerrit.wikimedia.org/r/574765

Change 574766 merged by Filippo Giunchedi:
[operations/puppet@production] puppetboard: log to logging pipeline

https://gerrit.wikimedia.org/r/574766

Mentioned in SAL (#wikimedia-operations) [2020-02-25T16:17:45Z] <godog> restart debmonitor / puppetboard - T245512

Change 573937 abandoned by Filippo Giunchedi:
service: logging pipeline support for uwsgi

Reason:
See related code reviews in T245512 instead

https://gerrit.wikimedia.org/r/573937