Followup from T141270, should setup an ELK stack that is accessible only to admins and has all the logs from syslog.
This is less prioritary now that we have prometheus with alerts (the original tasks had a side of alerting on them), would still be useful to have a central place to search for logs.
One log to rule them all.
It would be good to have logstash for at least tools-ops logs, which includes
- basic system logging (dmesg / syslog)
- infra logging (puppet, apt)
- mail (exim on tools-mail)
- k8s?
- bigbrother actions
- Redis?
- ssh (also useful to help people with issues logging in)