Page MenuHomePhabricator
Create Task
Maniphest T222356

facter3: Unable to parse routing table
Closed, ResolvedPublic
Actions

Description

It was nboticed in T222326 that facter3 is having problems parsing the routng table on some systems namly cp* systems. This causes facter to emit lots of warning messages when ever it is called, including via puppet . FACT-1394 seems related

cp3041 ~ % sudo puppet agent -t                                                             [10:23:31]
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.0.130 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.0.132 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.16.22 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.16.24 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.32.67 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.32.69 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.48.101 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.48.103 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.0.122 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.0.125 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.0.127 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.16.133 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.16.136 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.16.138 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.32.112 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.32.115 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.48.23 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.48.27 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:101:10:192:0:122 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:101:10:192:0:125 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:101:10:192:0:127 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:102:10:192:16:133 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:102:10:192:16:136 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:102:10:192:16:138 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:103:10:192:32:112 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:103:10:192:32:115 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:104:10:192:48:23 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:104:10:192:48:27 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:101:10:64:0:130 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:101:10:64:0:132 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:102:10:64:16:22 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:102:10:64:16:24 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:103:10:64:32:67 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:103:10:64:32:69 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:107:10:64:48:101 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:107:10:64:48:103 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Downgrading to PSON for future requests
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.0.130 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.0.132 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.16.22 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.16.24 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.32.67 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.32.69 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.48.101 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.64.48.103 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.0.122 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.0.125 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.0.127 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.16.133 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.16.136 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.16.138 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.32.112 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.32.115 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.48.23 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '10.192.48.27 via 10.20.0.1 dev eno1  mtu lock 1450'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:101:10:192:0:122 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:101:10:192:0:125 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:101:10:192:0:127 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:102:10:192:16:133 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:102:10:192:16:136 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:102:10:192:16:138 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:103:10:192:32:112 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:103:10:192:32:115 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:104:10:192:48:23 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:860:104:10:192:48:27 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:101:10:64:0:130 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:101:10:64:0:132 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:102:10:64:16:22 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:102:10:64:16:24 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:103:10:64:32:67 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:103:10:64:32:69 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:107:10:64:48:101 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Warning: Facter: Could not process routing table entry: Expected a destination followed by key/value pairs, got '2620:0:861:107:10:64:48:103 via fe80::1 dev eno1 metric 1024  mtu lock 1450 pref medium'
Info: Caching catalog for cp3041.esams.wmnet
Info: Applying configuration version '1556792620'
Notice: tlsproxy::localssl instance unified on port 443 with server name www.wikimedia.org is the default server.
Notice: /Stage[main]/Profile::Cache::Ssl::Unified/Tlsproxy::Localssl[unified]/Notify[tlsproxy localssl default_server on port 443]/message: defined 'message' as 'tlsproxy::localssl instance unified on port 443 with server name www.wikimedia.org is the default server.'
Notice: Applied catalog in 39.27 seconds

Related Objects
Search...

Event Timeline

jbond triaged this task as Medium priority.May 2 2019, 10:25 AM
jbond created this task.
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMay 2 2019, 10:25 AM
jbond added a subscriber: ayounsi.May 2 2019, 2:27 PM

This is due to a bug in facter fundamentally cased because there are an even number or words in the output of ip route show. I have proposed a fix upstream and i believe the warning is harmless (in fact my PR removes the warning all together). If we want to work around this issue more we have a number of options

  • patch or facter package (@MoritzMuehlenhoff opinion?)
  • update modules/interface/manifests/route.pp to remove the lock keyword. lock stops the kernel from updating the MTU via PTMU. I'm unsure if that may cause issues @ayounsi could likely add more information here?
ayounsi added a comment.May 2 2019, 3:02 PM

The lock is needed to workaround a bug with the kernel/ipsec.
See https://gerrit.wikimedia.org/r/c/operations/puppet/+/437784 and https://phabricator.wikimedia.org/T195365

• ema subscribed.May 2 2019, 3:25 PM
jbond added a comment.May 2 2019, 3:26 PM
In T222356#5153203, @ayounsi wrote:

The lock is needed to workaround a bug with the kernel/ipsec.
See https://gerrit.wikimedia.org/r/c/operations/puppet/+/437784 and https://phabricator.wikimedia.org/T195365

Ahh ok that makes sense thanks

MoritzMuehlenhoff added a comment.May 6 2019, 2:16 PM
In T222356#5153037, @jbond wrote:

This is due to a bug in facter fundamentally cased because there are an even number or words in the output of ip route show. I have proposed a fix upstream and i believe the warning is harmless (in fact my PR removes the warning all together). If we want to work around this issue more we have a number of options

I think we should patch facter once your PR is reviewed/merged upstream to address this for good. But I think it's fine to proceed with the facter rollout given that this is harmless log spam and only affects ~ 10% of our servers (mc and cp hosts).

jbond added a comment.May 7 2019, 11:10 AM

I think we should patch facter once your PR is reviewed/merged upstream to address this for good. But I think it's fine to proceed with the facter rollout given that this is harmless log spam and only affects ~ 10% of our servers (mc and cp hosts).

Sounds good to me i will proceed with the upgrade.

jbond added a comment.May 8 2019, 4:37 PM
This comment was removed by jbond.
jbond added a comment.May 8 2019, 4:38 PM

correct pull request https://github.com/puppetlabs/facter/pull/1775

jbond added a comment.Jun 28 2019, 2:07 PM

The

In T222356#5167750, @jbond wrote:

correct pull request https://github.com/puppetlabs/facter/pull/1775

This has now been merged and i have build a new package [avalible in cp1078:~jbond/] which is currently running on cp1076. I leave this ruining over the weekend. Assuming we don't see any other regressions and pending a review from @MoritzMuehlenhoff to ensure i have built the package correctly we should be able to roll out on Monday.

MoritzMuehlenhoff added a comment.Jul 1 2019, 10:37 AM

The updated package in boron:~jbond/src/facter-3.11.0 looks good. IMO we can ignore to backport this to jessie? the only cp host on jessie is the obsolete cp1008 which will be migrated to stretch/new server soon.

One thing to consider for buster: So far we've used the facter version in buster, so I think we have two options:

  1. We extend the component also to buster and adapt the puppet logic to pull it in
  2. The patch is merged upstream, but is sufficiently noisy to be eligible for a buster point update (the time has passed for Buster 10.0). We can also reach out to the Puppet maintainers in Debian and propose to fix this in Buster 10.1 and not upload a custom version for buster-wikimedia (as we don't run any caches on buster yet)
jbond added a comment.Jul 1 2019, 10:56 AM

One thing to consider for buster: So far we've used the facter version in buster, so I think we have two options:

I would prefer to go for option two until this is actually an issue

• ema added a comment.Jul 1 2019, 11:56 AM
In T222356#5295683, @MoritzMuehlenhoff wrote:

The updated package in boron:~jbond/src/facter-3.11.0 looks good. IMO we can ignore to backport this to jessie? the only cp host on jessie is the obsolete cp1008 which will be migrated to stretch/new server soon.

Yes, no need to bother with a jessie backport.

jbond added a comment.Jul 1 2019, 12:16 PM

The update has now been rolled out.

MoritzMuehlenhoff added a comment.Jul 1 2019, 12:57 PM
In T222356#5295760, @jbond wrote:

One thing to consider for buster: So far we've used the facter version in buster, so I think we have two options:

I would prefer to go for option two until this is actually an issue

I'll file a bug

In T222356#5295760, @jbond wrote:

One thing to consider for buster: So far we've used the facter version in buster, so I think we have two options:

I would prefer to go for option two until this is actually an issue

I'll file a bug in Debian for this (as it's unfixed in buster/sid currently)

MoritzMuehlenhoff added a comment.Jul 4 2019, 1:03 PM

There was a pre-existing bugreport in Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=918250 I've followed up there along with the offer to help get that fixed in a buster point release.

Stashbot added a comment.Sep 9 2019, 1:31 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-09T13:31:09Z] <moritzm> installing facter update from buster 10.1 point release (T222356)

MoritzMuehlenhoff added a comment.Sep 9 2019, 1:51 PM
The fix landed in Buster 10.1 and I've rolled it out to our Buster hosts. I think we can close this task.
MoritzMuehlenhoff closed this task as Resolved.Sep 9 2019, 2:09 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits