Change Details

We are getting a lot of cron spam from the "smart-data-dump" cron job due to: ``` WARN puppetlabs.facter - Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.0.131 via 10.20.0.1 dev eno1 mtu lock 1450' ``` The script is `./modules/smart/files/smart-data-dump` and it has been added in T86552. Currently and at first, only cp* hosts were affected. The root cause is a bug in facter which the script calls in: ` raw_output = subprocess.check_output(['/usr/bin/facter', '--puppet', '--json', fact_name])` When running facter with -d to get debug output it can be see that it runs `ip show route` and tries to parse the output: `DEBUG leatherman.execution:93 - executing command: /sbin/ip route show` which then throws a lot of warnings like: `[cp3041:~] $ sudo facter --puppet --json raid | grep WARN WARN puppetlabs.facter - Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:107:10:64:48:101 via fe80::1 dev eno1 metric 1024 mtu lock 1450 pref medium'` This is very similar, but not identical, to the upstream bug https://tickets.puppetlabs.com/browse/FACT-1394 In that bug the parsing fails when there is "linkdown" in the ip route output, but that isn't the case for us. An attempt was made to add `-l error` to the facter command in order to set the loglevel to error and suppress the warnings to stop the cron spam. https://gerrit.wikimedia.org/r/c/operations/puppet/+/507634 While this worked fine on the cp* servers displaying the issue..after merging it caused even more and new cron spam on non-cp* hosts. The reason for that were different facter versions. For some reason cp* hosts appear to have facter 3.x while most other hosts have facter 2.x, even when both are on stretch. In facter 2.x the "-l error" option does not exist which lead to the new spam from these hosts. So that change was reverted and now we are back to the original state.. cp* hosts are affected but others are not. Also the cp* hosts show the same warnings on each puppet run on the console.

We are getting a lot of cron spam from the "smart-data-dump" cron job. The script is `./modules/smart/files/smart-data-dump` and it has been added in T86552. Currently and at first, only cp* hosts were affected. The root cause is a bug in facter which the script calls in: ` raw_output = subprocess.check_output(['/usr/bin/facter', '--puppet', '--json', fact_name])` When running facter with -d to get debug output it can be see that it runs `ip show route` and tries to parse the output: `DEBUG leatherman.execution:93 - executing command: /sbin/ip route show` which then throws a lot of warnings like: ``` [cp3041:~] $ sudo facter --puppet --json raid | grep WARN WARN puppetlabs.facter - Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:107:10:64:48:101 via fe80::1 dev eno1 metric 1024 mtu lock 1450 pref medium' ``` This is very similar, but not identical, to the upstream bug https://tickets.puppetlabs.com/browse/FACT-1394 In that bug the parsing fails when there is "linkdown" in the ip route output, but that isn't the case for us. An attempt was made to add `-l error` to the facter command in order to set the loglevel to error and suppress the warnings to stop the cron spam. https://gerrit.wikimedia.org/r/c/operations/puppet/+/507634 While this worked fine on the cp* servers displaying the issue..after merging it caused even more and new cron spam on non-cp* hosts. The reason for that were different facter versions. For some reason cp* hosts appear to have facter 3.x while most other hosts have facter 2.x, even when both are on stretch. In facter 2.x the "-l error" option does not exist which lead to the new spam from these hosts. So that change was reverted and now we are back to the original state.. cp* hosts are affected but others are not. Also the cp* hosts show the same warnings on each puppet run on the console.

We are getting a lot of cron spam from the "smart-data-dump" cron job due to: ``` WARN puppetlabs.facter - Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.0.131 via 10.20.0.1 dev eno1 mtu lock 1450' ```. The script is `./modules/smart/files/smart-data-dump` and it has been added in T86552. Currently and at first, only cp* hosts were affected. The root cause is a bug in facter which the script calls in: ` raw_output = subprocess.check_output(['/usr/bin/facter', '--puppet', '--json', fact_name])` When running facter with -d to get debug output it can be see that it runs `ip show route` and tries to parse the output: `DEBUG leatherman.execution:93 - executing command: /sbin/ip route show` which then throws a lot of warnings like: ` ``` [cp3041:~] $ sudo facter --puppet --json raid | grep WARN WARN puppetlabs.facter - Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:107:10:64:48:101 via fe80::1 dev eno1 metric 1024 mtu lock 1450 pref medium'` ``` This is very similar, but not identical, to the upstream bug https://tickets.puppetlabs.com/browse/FACT-1394 In that bug the parsing fails when there is "linkdown" in the ip route output, but that isn't the case for us. An attempt was made to add `-l error` to the facter command in order to set the loglevel to error and suppress the warnings to stop the cron spam. https://gerrit.wikimedia.org/r/c/operations/puppet/+/507634 While this worked fine on the cp* servers displaying the issue..after merging it caused even more and new cron spam on non-cp* hosts. The reason for that were different facter versions. For some reason cp* hosts appear to have facter 3.x while most other hosts have facter 2.x, even when both are on stretch. In facter 2.x the "-l error" option does not exist which lead to the new spam from these hosts. So that change was reverted and now we are back to the original state.. cp* hosts are affected but others are not. Also the cp* hosts show the same warnings on each puppet run on the console.