Here's what the logs to disk look like:

root@mailman-mailman02:~# tree /var/log/mailman3/
/var/log/mailman3/
├── bounce.log
├── debug.log
├── mailman.log
├── mailman.log.1
├── mailman.log.2.gz
├── mailman.log.3.gz
├── mailman.log.4.gz
├── mailman.log.5.gz
├── plugins.log
├── smtp.log
└── web
    ├── mailman-web.log
    ├── mailman-web.log.1
    ├── mailman-web.log.2.gz
    ├── mailman-web.log.3.gz
    ├── mailman-web.log.4.gz
    └── mailman-web.log.5.gz

What's the preferred way to get these into logstash?

@lmata, we could use some help in setting this up.

Sure thing @Legoktm will discuss with team and share notes here

This is the last hard blocker we have for mailman3 at this point - is there a guide we can follow or an example we can copy/paste from?

I remember writing something for ores that sends all log entries to logstash (it was also uwsgi so it should be really similar) if you want to look it up but I'm not 100% sure we should send all uwsgi logs to logstash (I think it's possible to send >= WARN level logs to logstash only while logging in a lower level in file.). I can take a stab at it.

For daemons that are logging to syslog/journald the tl;dr to get the logs in logstash is to add the "program name" to modules/profile/files/rsyslog/lookup_table_output.json with value kafka local (or only kafka if you are not interested in local logs). For daemons logging to local files, tl;dr similar setup plus the "input file" part of rsyslog (i.e. rsyslog::input::file). Hope that helps! Happy to review patches of course and/or provide more guidance

Change 681648 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/puppet@production] rsyslog: Add mailman3 to list of accepted daemons

https://gerrit.wikimedia.org/r/681648

This is for mailman3 process. I haven't added mailman3web yet.

PCC is noop: https://puppet-compiler.wmflabs.org/compiler1001/727/lists1002.wikimedia.org/index.html 🤔

Bizarre PCC is a NOOP indeed. The patch LGTM, but I see mailman3 didn't log anything to journald on lists1002 since this morning?

it's a test system atm so not super active right now. I can send some mails to trigger some logs if you want to.

Change 682736 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/puppet@production] lists: Send error logs of apache2/exim4 to logstash

https://gerrit.wikimedia.org/r/682736

Change 682737 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/puppet@production] mailman3: Increase the log level to WARNING and send them to logstash

https://gerrit.wikimedia.org/r/682737

Change 681648 abandoned by Ladsgroup:

[operations/puppet@production] rsyslog: Add mailman3 to list of accepted daemons

Reason:

We decided not to collect logs in logstash due to sensitive nature of such logs.

https://gerrit.wikimedia.org/r/681648

Change 682737 abandoned by Ladsgroup:

[operations/puppet@production] mailman3: Increase the log level to WARNING and send them to logstash

Reason:

We decided against this.

https://gerrit.wikimedia.org/r/682737

Change 682736 abandoned by Ladsgroup:

[operations/puppet@production] lists: Send error logs of apache2/exim4 to logstash

Reason:

We decided against this.

https://gerrit.wikimedia.org/r/682736