Out of 1757 servers using rsyslogd, 230 fail validating their configuration with the following command:
$ /usr/sbin/rsyslogd -N1 -f /etc/rsyslog.confThe vast majority of those errors are due to duplicate module loading:
(143) cloudelastic[1001-1006].wikimedia.org,debmonitor2002.codfw.wmnet,debmonitor1002.eqiad.wmnet,elastic[2025-2060].codfw.wmnet,elastic[1032-1038,1040-1067].eqiad.wmnet,netbox[1001,2001].wikimedia.org,netbox-dev2001.wikimedia.org,ores[2001-2009].codfw.wmnet,ores[1001-1009].eqiad.wmnet,puppetboard2001.codfw.wmnet,puppetboard1001.eqiad.wmnet,relforge[1003-1004].eqiad.wmnet,sessionstore[2001-2003].codfw.wmnet,sessionstore[1001-1003].eqiad.wmnet,thumbor[2001-2004].codfw.wmnet,thumbor[1001-1004].eqiad.wmnet,wcqs[2001-2003].codfw.wmnet,wcqs[1001-1003].eqiad.wmnet,wdqs[2001-2008].codfw.wmnet,wdqs[1003-1013].eqiad.wmnet rsyslogd: module 'mmjsonparse' already in this config, cannot be added [v8.190 1.0 try https://www.rsyslog.com/e/2221 ] rsyslogd: module 'omkafka' already in this config, cannot be added [v8.1901.0 try https://www.rsyslog.com/e/2221 ]
And:
(57) aqs[1004-1015].eqiad.wmnet,maps[2005-2010].codfw.wmnet,maps[1005-1010].eqiad.wmnet,restbase[2009-2023].codfw.wmnet,restbase[1016-1030].eqiad.wmnet,restbase-dev[1004-1006].eqiad.wmnet: rsyslogd: module 'mmjsonparse' already in this config, cannot be added [v8.190 1.0 try https://www.rsyslog.com/e/2221 ] rsyslogd: module 'omkafka' already in this config, cannot be added [v8.1901.0 try https://www.rsyslog.com/e/2221 ] rsyslogd: module 'imudp' already in this config, cannot be added [v8.1901.0 try https://www.rsyslog.com/e/2221 ]
The error really should just be a warning in my opinion, given that rsyslog seems to be working just fine after those. To verify the assumption, I created this config snippet:
# omkafka module already loaded by previous configuration files module(load="omkafka") if $programname == 'ematest' then /var/log/ematest.log
I then proceeded restarting rsyslogd and got an error message as expected:
Sep 30 07:16:21 cptext rsyslogd[29545]: module 'omkafka' already in this config, cannot be added [v8.1901.0 try https://www.rsyslog.com/e/2221 ]
rsyslogd started just fine (see T290870). I then logged an entry to syslog and verified that, despite the error, the code following my duplicate include still worked:
$ sudo logger -p daemon.err "just a test $RANDOM" -t ematest $ sudo tail /var/log/ematest.log Sep 30 11:36:03 cptext ematest: just a test 10312
Conversely, introducing a syntax error in a snippet loaded before my test config causes rsyslog to start as if everything went fine, but no logs end up under ematest.log when trying the commands above. This is the behavior described in T290870, and we really would like to catch those situations. For this reason, even though duplicate includes seem harmless, we want to get rid of them so that we can validate rsyslogd's configuration with AbortOnUncleanConfig.
Unfortunately, rsyslog does not have a function to check whether or not a module has been loaded already, ie: we cannot do something like this:
if module_loaded("omkafka") then
module(load="omkafka")We could either (1) patch rsyslog to consider "double-inclusion" as a warning instead of an error, or (2) avoid including modules twice. The simplest way to achieve (2) which comes to mind is adding a config snippet like this to the 200 servers mentioned earlier in this task, and avoiding all other module includes:
# /etc/rsyslog.d/00-load-modules.conf module(load="omkafka") module(load="mmjsonparse") module(load="imudp")