Page MenuHomePhabricator

Move service::uwsgi logs to logging pipeline
Closed, ResolvedPublic

Description

uwsgi (via service::uwsgi) logs both to file and to logstash (json over udp) over the network with logger=logstash socket:logstash.svc.eqiad.wmnet:11514, we should move to logging to localhost instead (via profile::rsyslog::udp_json_logback_compat) or syslog directly, and from there onto the logging pipeline

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 18 2020, 1:49 PM
Volans added a subscriber: Volans.Feb 18 2020, 2:15 PM

@fgiunchedi what's the current best practice here? Debmonitor is just using service::uwsgi that automatically logs to /srv/log/debmonitor/main.log and AFAIK doesn't log to the local syslog normal connections but just restarts of the daemon.

I guess we would like to use a shared solution for all the uwsgi servers we have.

@fgiunchedi what's the current best practice here? Debmonitor is just using service::uwsgi that automatically logs to /srv/log/debmonitor/main.log and AFAIK doesn't log to the local syslog normal connections but just restarts of the daemon.

Essentially two options: json to syslog/journald (works best on Buster, before that journald has max 2k line length limitation) or json to local udp syslog (via rsyslog and '::profile::rsyslog::udp_json_logback_compat'). For a standard solution that applies to all uwsgi servers I'm thinking probably the latter is the easiest ATM (alternatively wait for Buster migration), what do you think ?

I suggest that we implement the logic in service::uwsgi allowing for an easy opt-in from all the users of that define, that I bet is used on WMCS too.
For the implementation between the option if we have time probably nicer to implement directly the buster solution and apply it while migrating to it.
But if we want/need to push for a quicker migration then let's do the interim local udp that is compatible with stretch too.

Agreed, I'll send out patches to add switches to service::uwsgi for logging pipeline. I agree the optimal would be journald although I suspect the full Buster migration isn't imminent at this point, will try with rsyslog first.

Change 573936 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] service: fix uwsgi logstash_port_logback

https://gerrit.wikimedia.org/r/573936

Change 573937 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] service: logging pipeline support for uwsgi

https://gerrit.wikimedia.org/r/573937

fgiunchedi renamed this task from Move debmonitor uwsgi logs to logging pipeline to Move service::uwsgi logs to logging pipeline.Feb 21 2020, 9:25 AM
fgiunchedi updated the task description. (Show Details)

Change 573936 merged by Filippo Giunchedi:
[operations/puppet@production] service: fix uwsgi logstash_port_logback

https://gerrit.wikimedia.org/r/573936

Change 574765 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] debmonitor: log to logging pipeline

https://gerrit.wikimedia.org/r/574765

Change 574766 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] puppetboard: log to logging pipeline

https://gerrit.wikimedia.org/r/574766

Change 574765 merged by Filippo Giunchedi:
[operations/puppet@production] debmonitor: log to logging pipeline

https://gerrit.wikimedia.org/r/574765

Change 574766 merged by Filippo Giunchedi:
[operations/puppet@production] puppetboard: log to logging pipeline

https://gerrit.wikimedia.org/r/574766

Mentioned in SAL (#wikimedia-operations) [2020-02-25T16:17:45Z] <godog> restart debmonitor / puppetboard - T245512

Change 573937 abandoned by Filippo Giunchedi:
service: logging pipeline support for uwsgi

Reason:
See related code reviews in T245512 instead

https://gerrit.wikimedia.org/r/573937

fgiunchedi closed this task as Resolved.Feb 25 2020, 4:50 PM
fgiunchedi claimed this task.

All service::uwsgi roles now log to the logging pipeline!