Since 2020/08/05 at ~16 it seems rsyslog has started failing on centrallog hosts, e.g.
Aug 06 08:27:25 centrallog1001 rsyslogd[28353]: action 'fwd_centrallog2001.codfw.wmnet:6514' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. Aug 06 08:27:31 centrallog1001 rsyslogd[28353]: main Q:Reg: high activity - starting 1 additional worker thread(s), currently 1 active worker threads. [v8.1901.0 try https://www.rsyslog.com/e/2439 ] Aug 06 08:27:31 centrallog1001 rsyslog_exporter[28354]: 2020/08/06 08:27:31 input ended, exiting normally Aug 06 08:27:31 centrallog1001 systemd[1]: rsyslog.service: Main process exited, code=killed, status=11/SEGV Aug 06 08:27:31 centrallog1001 systemd[1]: rsyslog.service: Failed with result 'signal'. Aug 06 08:27:31 centrallog1001 systemd[1]: rsyslog.service: Service RestartSec=100ms expired, scheduling restart. Aug 06 08:27:31 centrallog1001 systemd[1]: rsyslog.service: Scheduled restart job, restart counter is at 13. Aug 06 08:27:31 centrallog1001 systemd[1]: Stopped System Logging Service. Aug 06 08:27:31 centrallog1001 systemd[1]: Starting System Logging Service... Aug 06 08:27:31 centrallog1001 rsyslog_exporter[31213]: 2020/08/06 08:27:31 Listening on :9105 Aug 06 08:27:32 centrallog1001 rsyslogd[31212]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.1901.0] Aug 06 08:27:32 centrallog1001 rsyslogd[31212]: [origin software="rsyslogd" swVersion="8.1901.0" x-pid="31212" x-info="https://www.rsyslog.com"] start Aug 06 08:27:32 centrallog1001 systemd[1]: Started System Logging Service.
Looks like the latest rsyslog version is significantly more stable on centrallog1001! We'll be running 8.2008.0-1~bpo10+1 on centrallog hosts only then, implementation wise these are the steps:
- Upgrade centrallog2001
- Create an APT component for rsyslog
- Use the component on centrallog hosts only (os_version <= buster) and make sure the component's APT priorities are correct, so that on reimage the component's version is installed
- Upload rsyslog 8.2008.0-1~bpo10+1 and librelp 1.7.0-1~bpo10+1 to said component