Page MenuHomePhabricator

rsyslog errors about duplicate module includes
Open, MediumPublic

Description

Out of 1757 servers using rsyslogd, 230 fail validating their configuration with the following command:

$ /usr/sbin/rsyslogd -N1 -f /etc/rsyslog.conf

The vast majority of those errors are due to duplicate module loading:

(143) cloudelastic[1001-1006].wikimedia.org,debmonitor2002.codfw.wmnet,debmonitor1002.eqiad.wmnet,elastic[2025-2060].codfw.wmnet,elastic[1032-1038,1040-1067].eqiad.wmnet,netbox[1001,2001].wikimedia.org,netbox-dev2001.wikimedia.org,ores[2001-2009].codfw.wmnet,ores[1001-1009].eqiad.wmnet,puppetboard2001.codfw.wmnet,puppetboard1001.eqiad.wmnet,relforge[1003-1004].eqiad.wmnet,sessionstore[2001-2003].codfw.wmnet,sessionstore[1001-1003].eqiad.wmnet,thumbor[2001-2004].codfw.wmnet,thumbor[1001-1004].eqiad.wmnet,wcqs[2001-2003].codfw.wmnet,wcqs[1001-1003].eqiad.wmnet,wdqs[2001-2008].codfw.wmnet,wdqs[1003-1013].eqiad.wmnet
 rsyslogd: module 'mmjsonparse' already in this config, cannot be added  [v8.190 1.0 try https://www.rsyslog.com/e/2221 ]
 rsyslogd: module 'omkafka' already in this config, cannot be added  [v8.1901.0 try https://www.rsyslog.com/e/2221 ]

And:

(57) aqs[1004-1015].eqiad.wmnet,maps[2005-2010].codfw.wmnet,maps[1005-1010].eqiad.wmnet,restbase[2009-2023].codfw.wmnet,restbase[1016-1030].eqiad.wmnet,restbase-dev[1004-1006].eqiad.wmnet:
rsyslogd: module 'mmjsonparse' already in this config, cannot be added  [v8.190 1.0 try https://www.rsyslog.com/e/2221 ]
rsyslogd: module 'omkafka' already in this config, cannot be added  [v8.1901.0 try https://www.rsyslog.com/e/2221 ]
rsyslogd: module 'imudp' already in this config, cannot be added  [v8.1901.0 try https://www.rsyslog.com/e/2221 ]

The error really should just be a warning in my opinion, given that rsyslog seems to be working just fine after those. To verify the assumption, I created this config snippet:

# omkafka module already loaded by previous configuration files
module(load="omkafka")

if $programname == 'ematest' then /var/log/ematest.log

I then proceeded restarting rsyslogd and got an error message as expected:

Sep 30 07:16:21 cptext rsyslogd[29545]: module 'omkafka' already in this config, cannot be added  [v8.1901.0 try https://www.rsyslog.com/e/2221 ]

rsyslogd started just fine (see T290870). I then logged an entry to syslog and verified that, despite the error, the code following my duplicate include still worked:

$ sudo logger -p daemon.err "just a test $RANDOM" -t ematest
$ sudo tail /var/log/ematest.log 
Sep 30 11:36:03 cptext ematest: just a test 10312

Conversely, introducing a syntax error in a snippet loaded before my test config causes rsyslog to start as if everything went fine, but no logs end up under ematest.log when trying the commands above. This is the behavior described in T290870, and we really would like to catch those situations. For this reason, even though duplicate includes seem harmless, we want to get rid of them so that we can validate rsyslogd's configuration with AbortOnUncleanConfig.

Unfortunately, rsyslog does not have a function to check whether or not a module has been loaded already, ie: we cannot do something like this:

if module_loaded("omkafka") then
    module(load="omkafka")

We could either (1) patch rsyslog to consider "double-inclusion" as a warning instead of an error, or (2) avoid including modules twice. The simplest way to achieve (2) which comes to mind is adding a config snippet like this to the 200 servers mentioned earlier in this task, and avoiding all other module includes:

# /etc/rsyslog.d/00-load-modules.conf
module(load="omkafka")
module(load="mmjsonparse")
module(load="imudp")

Event Timeline

ema triaged this task as Medium priority.Oct 8 2021, 11:54 AM

Change 761455 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] rsyslog: add 00-load_modules.conf

https://gerrit.wikimedia.org/r/761455

Change 761455 merged by Herron:

[operations/puppet@production] rsyslog: add 00-load_modules.conf

https://gerrit.wikimedia.org/r/761455

herron added a subscriber: herron.

To complicate matters, rsyslog also appears to throw errors when a module is loaded but not actively used, e.g.:

$ /usr/sbin/rsyslogd -N1 -f /etc/rsyslog.conf
rsyslogd: version 8.1901.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: imudp: module loaded, but no listeners defined - no input will be gathered [v8.1901.0 try https://www.rsyslog.com/e/2212 ]

So, looks like a realm wide 00-load_modules.conf is out.

The next idea that comes to mind is using puppet to include rsyslog module loads in a way that will do the right thing when called multiple times.

Change 766789 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] Revert \"rsyslog: add 00-load_modules.conf\"

https://gerrit.wikimedia.org/r/766789

Change 766789 merged by Herron:

[operations/puppet@production] Revert \"rsyslog: add 00-load_modules.conf\"

https://gerrit.wikimedia.org/r/766789

Change 766814 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] rsyslog: create rsyslog::module for easy module loading

https://gerrit.wikimedia.org/r/766814