Page MenuHomePhabricator

puppet populate failing on some nodes
Open, Needs TriagePublic

Description

the puppet-populate script and the puppet lookup --compile fail to resolve trusted facts. This means that it is not possible to use the compile command and also means some nodes are not present in the PCC .

PUP-5441 and PUP-9207 both seems to document similar issues

populate_puppetdb.py shows the following error when compiling however it only seems to trigger on some nodes and im not currently sure why

 2020-03-20T13:00:52 ] ERROR: Compilation failed for host chlorine.eqiad.wmnet: Command '['puppet', 'master', '--vardir=/var/lib/catalog-differ/puppet', '--modulepath=/tmp/fill-puppetdblS5HL2/1/production/private/modules:/tmp/fill-puppetdblS5HL2/1/production/src/modules', '--confdir=/tmp/fill-puppetdblS5HL2/1/production/src', '--compile=chlorine.eqiad.wmnet', '--color=false', '--yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet', '--manifest=$confdir/manifests', '--environmentpath=$confdir/environments', '--storeconfigs', '--storeconfigs_backend=puppetdb']' returned non-zero exit status 30
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/puppet_compiler/puppet.py", line 83, in compile_storeconfigs
    subprocess.check_call(cmd, stdout=out, stderr=err, env=env)
  File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['puppet', 'master', '--vardir=/var/lib/catalog-differ/puppet', '--modulepath=/tmp/fill-puppetdblS5HL2/1/production/private/modules:/tmp/fill-puppetdblS5HL2/1/production/src/modules', '--confdir=/tmp/fill-puppetdblS5HL2/1/production/src', '--compile=chlorine.eqiad.wmnet', '--color=false', '--yamldir=/var/lib/catalog-differ/puppet/yaml/puppetmaster1001.eqiad.wmnet', '--manifest=$confdir/manifests', '--environmentpath=$confdir/environments', '--storeconfigs', '--storeconfigs_backend=puppetdb']' returned non-zero exit status 30
Error: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /tmp/fill-puppetdblS5HL2/1/production/src/manifests/realm.pp, line: 23, column: 4) on node chlorine.eqiad.wmnet

Error: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /tmp/fill-puppetdblS5HL2/1/production/src/manifests/realm.pp, line: 23, column: 4) on node chlorine.eqiad.wmnet

Warning: Failed to compile catalog for node chlorine.eqiad.wmnet: Evaluation Error: Left match operand must result in a String value. Got an Undef Value. (file: /tmp/fill-puppetdblS5HL2/1/production/src/manifests/realm.pp, line: 23, column: 4) on node chlorine.eqiad.wmnet

Error: Failed to compile catalog for node chlorine.eqiad.wmnet: undefined method `to_resource' for nil:NilClass

Details

Related Gerrit Patches:

Event Timeline

jbond created this task.Fri, Mar 20, 1:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFri, Mar 20, 1:05 PM
jbond added a comment.Fri, Mar 20, 1:59 PM

I think the reason that it works on some nodes and not others is because the puppetdb contains trusted facts. My working theory is that if you successfully compile a host then puppet db will automaticity populate the trusted facts variable. Theses trusted facts will then be available on further compilations. Once trusted facts where added to realm.pp any nodes that were not already in the puppetdb started to fail compilation as the trusted facts are not available. however anything which was already in the DB would work successfully

Change 582048 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] realm.pp: fact the trusted certname fact if performing a lookup or pcc

https://gerrit.wikimedia.org/r/582048

jbond added a comment.Fri, Mar 20, 2:27 PM

looks like puppet lookup has plans to support this at some point

jbond moved this task from Unsorted 💣 to Active 🚁 on the User-jbond board.Tue, Mar 24, 1:29 PM

Change 582048 merged by Jbond:
[operations/puppet@production] realm.pp: trusted facts unavailable when performing a lookup or pcc

https://gerrit.wikimedia.org/r/582048