labtest puppetmaster is not working for clients
Closed, ResolvedPublic
Actions

Description

labtest images seem to be using puppmaster fqdn instead of service url. I think in the course of T171494: Refactor OpenStack Puppet to account for Neutron this was made consistent on the master side potentially and now images in labtest do not work. This may require regenerating or fixing the master side cert too.

From an example host in prod:

[agent]
server = labs-puppetmaster.wikimedia.org

From an example host using current images in labtest

[agent]
server = labtestpuppetmaster2001.wikimedia.org

I believe this results in:

2017-11-28T15:26:50.954661+00:00 labtestv3-autosign-true puppet-agent[1661]: (/File[/var/lib/puppet/lib]) Could not evaluate: Could not retrieve file metadata for puppet://labtestpuppetmaster2001.wikimedia.org/plugins: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=labtestpuppetmaster2001.wikimedia.org]

Details

Project	Branch	Lines +/-	Subject
operations/puppet	production	+10 -0	labs_bootstrapvz: firstboot.sh: bring back some resolv.conf magic
operations/puppet	production	+3 -6	bootstrapvz first boot: Go back to using named 'puppet' puppetmaster name
operations/puppet	production	+8 -1	wmcs puppetmasters: make cert_secret_path configurable per deploy
operations/puppet	production	+6 -2	labs_bootstrapvz: address labtest issues
operations/puppet	production	+2 -1	labtestpuppetmaster: further attempts to set the ca_server
operations/puppet	production	+1 -0	labtestpuppetmaster: define 'ca_server'
operations/puppet	production	+6 -8	Labtest: try to standardize on a service name for the puppetmaster

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T53494 Use Beta cluster as a true canary for code deployments (epic)
Open	None	T87220 Minimize infrastructure differences between Beta Cluster and production
Open	None	T196662 Set up LVS in beta like prod
Resolved	bd808	T166396 Program 1 Outcome 4: VPS hosting
Resolved	None	T167293 Nova-network to Neutron migration
Resolved	aborrero	T181523 labtest puppetmaster is not working for clients

Event Timeline

• chasemp created this task.Nov 28 2017, 4:22 PM

• chasemp assigned this task to aborrero.Apr 9 2018, 12:56 PM

New base images just use the generic 'puppet' name for the puppetmaster. The individual dns recursors alias 'puppet' to the proper puppetmaster.

bd808 added a project: cloud-services-team (Kanban).Apr 9 2018, 9:27 PM

ayounsi removed a subscriber: ayounsi.Apr 9 2018, 9:29 PM

aborrero mentioned this in T191814: labtest: instance creation via horizon.Apr 10 2018, 8:49 AM

the labtestpuppetmaster1001.wikimedia.org server might not be using the right certificate:

root@labtestpuppetmaster2001:~# grep ssl /etc/puppet/puppet.conf
ssldir = /var/lib/puppet/ssl
ssldir = /var/lib/puppet/server/ssl/
hostcert = /var/lib/puppet/server/ssl/certs/labtestpuppetmaster2001.wikimedia.org.pem
hostprivkey = /var/lib/puppet/server/ssl/private_keys/labtestpuppetmaster2001.wikimedia.org.pem

However:

root@labtestpuppetmaster2001:~# ls -la /var/lib/puppet/server/ssl/certs/
total 16
drwxr-xr-x 2 puppet puppet 4096 Apr  4 20:54 .
drwxrwx--x 9 puppet root   4096 Apr  3 17:23 ..
-rw-r--r-- 1 puppet puppet 1923 Apr  3 17:15 ca.pem
-rw-r----- 1 puppet puppet 1903 Apr  3 17:21 puppet.pem

and

root@labtestpuppetmaster2001:~# ls -la /var/lib/puppet/server/ssl/private_keys/
total 12
drwxr-x--- 2 puppet puppet 4096 Apr  3 17:21 .
drwxrwx--x 9 puppet root   4096 Apr  3 17:23 ..
-rw-r----- 1 puppet puppet 3243 Apr  3 17:21 puppet.pem

I would say puppet.pem is the certificate we should use:

root@labtestpuppetmaster2001:~# openssl x509 -in /var/lib/puppet/server/ssl/certs/puppet.pem -noout -subject -issuer -dates
subject= /CN=puppet
issuer= /CN=Puppet CA: palladium.eqiad.wmnet
notBefore=Jun 30 19:36:53 2015 GMT
notAfter=Jun 29 19:36:53 2020 GMT

My proposal is to switch /etc/puppet/puppet.conf to use:

hostcert = /var/lib/puppet/server/ssl/certs/puppet.pem
hostprivkey = /var/lib/puppet/server/ssl/private_keys/puppet.pem

After replacing the puppet.conf path to other certificates, I created some random VM with no luck:

[...]
2018-04-10T10:26:34.909082+00:00 t1 puppet-agent[1533]: Creating a new SSL key for t1.codfw.labtest
2018-04-10T10:26:35.471006+00:00 t1 puppet-agent[1533]: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
2018-04-10T10:26:35.474244+00:00 t1 puppet-agent[1533]: Creating a new SSL certificate request for t1.codfw.labtest
2018-04-10T10:26:35.507029+00:00 t1 puppet-agent[1533]: Certificate Request fingerprint (SHA256): 7B:55:72:BD:72:FE:77:68:7E:78:34:28:78:29:9F:30:11:B2:EE:00:2C:31:77:9D:8B:E4:12:95:44:98:7C:DC
2018-04-10T10:26:35.622137+00:00 t1 puppet-agent[1533]: Caching certificate for t1.codfw.labtest
2018-04-10T10:26:35.654981+00:00 t1 puppet-agent[1533]: Could not request certificate: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppet]
[...]

Anyway I see some communication with the puppetmaster:

Apr 10 10:26:20 labtestpuppetmaster2001 puppet-master[25414]: t1.andrewtestproject.codfw.labtest has a waiting certificate request
Apr 10 10:26:20 labtestpuppetmaster2001 puppet-master[25414]: Signed certificate request for t1.andrewtestproject.codfw.labtest
Apr 10 10:26:20 labtestpuppetmaster2001 puppet-master[25414]: Removing file Puppet::SSL::CertificateRequest t1.andrewtestproject.codfw.labtest at '/var/lib/puppet/server/ssl/ca/requests/t1.andrewtestproject.codfw.labtest.pem'
Apr 10 10:26:35 labtestpuppetmaster2001 puppet-master[25414]: t1.codfw.labtest has a waiting certificate request
Apr 10 10:26:36 labtestpuppetmaster2001 puppet-master[25414]: Signed certificate request for t1.codfw.labtest
Apr 10 10:26:36 labtestpuppetmaster2001 puppet-master[25414]: Removing file Puppet::SSL::CertificateRequest t1.codfw.labtest at '/var/lib/puppet/server/ssl/ca/requests/t1.codfw.labtest.pem'

I wonder if the base image contains the right CA certificate to validate the server (puppetmaster) certificate.

I wonder if the base image contains the right CA certificate to validate the server (puppetmaster) certificate.

That seems possible, although in theory the base image should contain no certificates at all and have totally fresh eyes for the puppetmaster.

That said, it might be worth trying to duplicate the server certs between labpuppetmaster1001 and labtestpuppetmaster2001, if they aren't already.

In T181523#4120747, @Andrew wrote:

That said, it might be worth trying to duplicate the server certs between labpuppetmaster1001 and labtestpuppetmaster2001, if they aren't already.

There are 2 different certs involved:

one in /etc/puppet/puppet.conf
one use in apache config

I'm not sure what is the used one, but apparently both deployments have the same 2.

What I'm trying now is:

I generated a debian-jessie-test.qcow2 image, with some tunning inside (I forked the manifest into /root/test-manifest.yaml and the firstboot.sh script into /root/test-firstboot.sh
I added a test user, passwd auth for testing purposes.
I'm adding the new image to glance, to lunch a test instance
I disabled the auto first puppet run, in the hope that I can jump into the machine and inspect what's wrong by running puppet agent by hand

aborrero merged a task: T191814: labtest: instance creation via horizon.Apr 11 2018, 11:35 AM

aborrero added a subscriber: aborrero.

I was finally able to jump by SSH into a test machine by means of:

following steps in previous comment.
identify VM disk image used by nova
copy elsewhere the image, and mount using guestfish. Push a root sshkey
copy back the disk image and re-start the VM
using a ssh config in my laptop like this:

## labtest
Host 10.196.*
    User root
    ProxyCommand ssh -W %h:%p labtestcontrol2001.wikimedia.org
    IdentityFile ~/.ssh/my_strong_key

Some of the tests I did, was to manually run openssl:

root@localhost:~# openssl s_client -connect puppet:8140
CONNECTED(00000003)
depth=0 CN = labtest-puppetmaster.wikimedia.org
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = labtest-puppetmaster.wikimedia.org
verify error:num=21:unable to verify the first certificate
verify return:1

It seems the self signed chain is not something that openssl accepts.

In case it matters, these are the involved DNS bits as seen from the VM point of view:

root@localhost:~# host puppet
puppet has address 208.80.153.108
puppet has address 208.80.153.108
puppet has address 208.80.153.108
root@localhost:~# host 208.80.153.108
108.153.80.208.in-addr.arpa domain name pointer labtestpuppetmaster2001.wikimedia.org.
root@localhost:~# host labtestpuppetmaster2001.wikimedia.org
labtestpuppetmaster2001.wikimedia.org has address 208.80.153.108
labtestpuppetmaster2001.wikimedia.org has IPv6 address 2620:0:860:4:208:80:153:108

Weird stuff. I'm trying to understand what is happening here, comparing VM<-->puppetmaster in labtest and main deployments.

[main] From the labpuppetmaster server point of view

The CA certificate in use seems to be the one at /var/lib/puppet/server/ssl/ca/ca_crt.pem:

aborrero@labpuppetmaster1001:~ $ sudo openssl x509 -in /var/lib/puppet/server/ssl/ca/ca_crt.pem -noout -fingerprint
SHA1 Fingerprint=A1:F4:B7:87:A7:89:34:57:DA:B6:B0:71:38:A1:D5:56:C7:E2:97:64
aborrero@labpuppetmaster1001:~ $ sudo openssl x509 -in /var/lib/puppet/server/ssl/ca/ca_crt.pem -noout -text
[...]
        Issuer: CN=Puppet CA: puppet
        Validity
            Not Before: Jul 26 15:17:59 2017 GMT
            Not After : Jul 26 15:17:59 2022 GMT
        Subject: CN=Puppet CA: puppet
[...]
            X509v3 Subject Key Identifier: 
                3C:98:64:31:78:C7:D8:0F:6F:FB:AA:15:6C:7D:63:3A:68:B4:DE:A9

[main] From a VM-connecting-to-labpuppetmaster point of view

a given VM has the right certificate (match #1), so SSL connection is OK (note how misleading the path is, since in the master this file contains a different CA certificate):

aborrero@stretch:~$ openssl x509 -in /usr/lib/ssl/certs/Puppet_Internal_CA.pem -noout -fingerprint
SHA1 Fingerprint=A1:F4:B7:87:A7:89:34:57:DA:B6:B0:71:38:A1:D5:56:C7:E2:97:64
aborrero@stretch:~$ openssl x509 -in labpuppetmaster_server_received.crt -noout -text
[...]
            X509v3 Authority Key Identifier: 
                keyid:3C:98:64:31:78:C7:D8:0F:6F:FB:AA:15:6C:7D:63:3A:68:B4:DE:A9
[...]
aborrero@stretch:~$ openssl x509 -in /usr/lib/ssl/certs/Puppet_Internal_CA.pem -noout -text
[...]
            X509v3 Subject Key Identifier: 
                3C:98:64:31:78:C7:D8:0F:6F:FB:AA:15:6C:7D:63:3A:68:B4:DE:A9
[...]

[labtest] From labtestpuppetmaster point of view

labtestpuppetmaster contains a different CA certificate:

aborrero@labtestpuppetmaster2001:~ $ sudo openssl x509 -in /var/lib/puppet/server/ssl/ca/ca_crt.pem -noout -fingerprint
SHA1 Fingerprint=B3:EA:E9:74:5F:D5:A8:92:4C:7A:3F:50:C7:DA:4B:B5:50:0B:FA:6E
aborrero@labtestpuppetmaster2001:~ $ sudo openssl x509 -in /var/lib/puppet/server/ssl/ca/ca_crt.pem -noout -text
[...]
        Issuer: CN=Puppet CA: puppet
        Validity
            Not Before: Apr  2 17:15:24 2018 GMT
            Not After : Apr  2 17:15:24 2023 GMT
        Subject: CN=Puppet CA: puppet
[...]
            X509v3 Subject Key Identifier: 
                D7:EE:E8:7C:B7:83:BA:61:05:33:A6:21:AB:D1:18:18:46:96:3D:18

[labtest] From VM-connecting-to-labtestpuppetmaster point of view

the server certificate we recv from the server is:

root@localhost:~# openssl x509 -in labtestpuppetmaster_server_received.crt-noout -text
[...]
        Issuer: CN=Puppet CA: puppet
        Validity
            Not Before: Sep 21 16:31:46 2017 GMT
            Not After : Sep 21 16:31:46 2022 GMT
        Subject: CN=labtest-puppetmaster.wikimedia.org
[...]
            X509v3 Subject Key Identifier: 
                BB:36:55:14:08:F4:FB:B4:D1:18:CF:41:1E:65:A9:90:0F:C5:EA:F9
            X509v3 Authority Key Identifier: 
                keyid:6B:8E:56:56:16:F0:8F:25:AA:AC:84:58:CB:F1:F5:4B:E6:95:FD:1F

the CA file doesn't match the certificate in #4:

root@localhost:~# openssl x509 -in /var/lib/puppet/ssl/certs/ca.pem -noout -fingerprint
SHA1 Fingerprint=B3:EA:E9:74:5F:D5:A8:92:4C:7A:3F:50:C7:DA:4B:B5:50:0B:FA:6E
root@localhost:~# openssl x509 -in /var/lib/puppet/ssl/certs/ca.pem -noout -text
[...]
        Issuer: CN=Puppet CA: puppet
        Validity
            Not Before: Apr  2 17:15:24 2018 GMT
            Not After : Apr  2 17:15:24 2023 GMT
        Subject: CN=Puppet CA: puppet
[...]
            X509v3 Subject Key Identifier: 
                D7:EE:E8:7C:B7:83:BA:61:05:33:A6:21:AB:D1:18:18:46:96:3D:18

questions and conclusions

Where did /var/lib/puppet/ssl/certs/ca.pem come from? (in labtest VM) It doesn't seem included in the bootstrapvz manifest.
We would need to find the CA with keyid:6B:8E:56:56:16:F0:8F:25:AA:AC:84:58:CB:F1:F5:4B:E6:95:FD:1F (missing) and inject that into the VM in labtest
If that CA is not found, we would need to recreate the server certificate and get it signed by CA with fingerprint B3:EA:E9:74:5F:D5:A8:92:4C:7A:3F:50:C7:DA:4B:B5:50:0B:FA:6E, which is the one included in the labtest VM.
There is no clear docs on this PKI layout on wikitech
The current layout is misleading and hard to debug. Certificates scattered in many different paths, paths which aren't the same in servers/clients.

Where did /var/lib/puppet/ssl/certs/ca.pem come from? (in labtest VM) It doesn't seem included in the bootstrapvz manifest

The firstboot script does this:

# Make sure nothing has leaked in certwise
rm -rf /var/lib/puppet/ssl/*

So I'm pretty sure that must be generated during the initial puppet run.

Ok, more tests. These are from a VM in labtest.

I'f I push the CA who signed the labtestpuppetmaster 'puppet' cert:

root@localhost:~# openssl s_client -connect puppet:8140 -servername puppet | grep verify
depth=0 CN = puppet
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = puppet
verify error:num=21:unable to verify the first certificate
verify return:1
    Verify return code: 21 (unable to verify the first certificate)
[...]

root@localhost:~# mv Puppet_Internal_CA.pem /etc/ssl/certs/Puppet_Internal_CA.pem
root@localhost:~# c_rehash
[...]

root@localhost:~# openssl s_client -connect puppet:8140 -servername puppet | grep verify
depth=1 CN = Puppet CA: puppet
verify return:1
depth=0 CN = puppet
verify return:1
[...]

Now openssl is able to establish the SSL connection.

But puppet agent is still failing. I guess puppet agent is just ignoring system certificates and using his own at /var/lib/puppet/ssl/certs?
If I put there the same CA file, puppet agent still fails, so not sure what's wrong here:

root@localhost:~# ls /var/lib/puppet/ssl/certs
ca.pem	localhost.codfw.labtest.pem  Puppet_Internal_CA.pem

root@localhost:~# puppet agent -t -v
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppet]
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppet]
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppet]
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppet]
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppet]
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppet]
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppet]

I turned on debug log in apache2 in labtestpuppetmaster. My testing client is 10.196.16.177 (file /var/log/apache2/error.log):

[Fri Apr 13 12:51:35.560439 2018] [ssl:info] [pid 2259] [client 10.196.16.177:50310] AH02008: SSL library error 1 in handshake (server labtest-puppetmaster.wikimedia.org:443)
[Fri Apr 13 12:51:35.560484 2018] [ssl:info] [pid 2259] SSL Library Error: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca (SSL alert number 48)
[Fri Apr 13 12:51:35.560495 2018] [ssl:info] [pid 2259] [client 10.196.16.177:50310] AH01998: Connection closed to child 1 with abortive shutdown (server puppet:443)
[Fri Apr 13 12:51:35.596588 2018] [ssl:info] [pid 2258] [client 10.196.16.177:50312] AH01964: Connection to child 0 established (server labtest-puppetmaster.wikimedia.org:443)
[Fri Apr 13 12:51:35.597962 2018] [ssl:debug] [pid 2258] ssl_engine_kernel.c(1908): [client 10.196.16.177:50312] AH02043: SSL virtual host for servername puppet found
[Fri Apr 13 12:51:35.610969 2018] [ssl:info] [pid 2258] [client 10.196.16.177:50312] AH02008: SSL library error 1 in handshake (server labtest-puppetmaster.wikimedia.org:443)
[Fri Apr 13 12:51:35.610992 2018] [ssl:info] [pid 2258] SSL Library Error: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca (SSL alert number 48)
[Fri Apr 13 12:51:35.610998 2018] [ssl:info] [pid 2258] [client 10.196.16.177:50312] AH01998: Connection closed to child 0 with abortive shutdown (server puppet:443)
[Fri Apr 13 12:51:35.639844 2018] [ssl:info] [pid 2261] [client 10.196.16.177:50314] AH01964: Connection to child 3 established (server labtest-puppetmaster.wikimedia.org:443)
[Fri Apr 13 12:51:35.641095 2018] [ssl:debug] [pid 2261] ssl_engine_kernel.c(1908): [client 10.196.16.177:50314] AH02043: SSL virtual host for servername puppet found
[Fri Apr 13 12:51:35.652260 2018] [ssl:info] [pid 2261] [client 10.196.16.177:50314] AH02008: SSL library error 1 in handshake (server labtest-puppetmaster.wikimedia.org:443)
[Fri Apr 13 12:51:35.652294 2018] [ssl:info] [pid 2261] SSL Library Error: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca (SSL alert number 48)
[Fri Apr 13 12:51:35.652303 2018] [ssl:info] [pid 2261] [client 10.196.16.177:50314] AH01998: Connection closed to child 3 with abortive shutdown (server puppet:443)
[Fri Apr 13 12:51:36.122662 2018] [ssl:info] [pid 2262] [client 208.80.153.74:49584] AH01964: Connection to child 4 established (server labtestpuppetmaster2001.wikimedia.org:8141)

It seems the puppet agent client asks first for labtest-puppetmaster.wikimedia.org as sever name? in tcp/443? unknown ca? weird data here.

I was able to get a puppet run almost working by sudo mv /var/lib/puppet/ssl/certs/Puppet_Internal_CA.pem /var/lib/puppet/ssl/certs/ca.pem.

The Puppet_Internal_CA.pem file is not in the VM (should be included) and the fingerprint is: SHA1 Fingerprint=A1:F4:B7:87:A7:89:34:57:DA:B6:B0:71:38:A1:D5:56:C7:E2:97:64

root@localhost:~# puppet agent -t -v
Info: Caching certificate_revocation_list for ca
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get certificate CRL for /CN=puppet]
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: /File[/var/lib/puppet/lib/dummy]/ensure: defined content as '{md5}d41d8cd98f00b204e9800998ecf8427e'
Notice: /File[/var/lib/puppet/lib/facter]/ensure: created
Notice: /File[/var/lib/puppet/lib/facter/apt.rb]/ensure: defined content as '{md5}3348ade54a8eb319f0d878f1c273785e'
Notice: /File[/var/lib/puppet/lib/facter/facter_dot_d.rb]/ensure: defined content as '{md5}d71e93183a680ac78bc0389fd50470a0'
Notice: /File[/var/lib/puppet/lib/facter/ganeti.rb]/ensure: defined content as '{md5}61b52f573b9c32853f6a3b16680029c8'
Notice: /File[/var/lib/puppet/lib/facter/initsystem.rb]/ensure: defined content as '{md5}5dd97dae4bdfe6a1d361b3be74d859fa'
Notice: /File[/var/lib/puppet/lib/facter/interface_primary.rb]/ensure: defined content as '{md5}8eb2423972cbd08ce2a4358f3d73aec0'
Notice: /File[/var/lib/puppet/lib/facter/ipmi.rb]/ensure: defined content as '{md5}69da6efc734cc00fbb1fcab0c1670c9d'
Notice: /File[/var/lib/puppet/lib/facter/lldp.rb]/ensure: defined content as '{md5}12af98d6ea0f2d420f4342bf9802df3e'
Notice: /File[/var/lib/puppet/lib/facter/lvm_support.rb]/ensure: defined content as '{md5}40cf6d0a36d9c3d3067809eb0950fc03'
Notice: /File[/var/lib/puppet/lib/facter/numa.rb]/ensure: defined content as '{md5}659fe95a64a8f4379110f1e7f1a3d9f6'
Notice: /File[/var/lib/puppet/lib/facter/package_provider.rb]/ensure: defined content as '{md5}539766a71dfb2f65e94a7c91bf413fcf'
Notice: /File[/var/lib/puppet/lib/facter/pe_version.rb]/ensure: defined content as '{md5}60d47406026c8201e51394227ddf780d'
Notice: /File[/var/lib/puppet/lib/facter/physicalcorecount.rb]/ensure: defined content as '{md5}9af631769ffd6924d56e14178ed0c43b'
Notice: /File[/var/lib/puppet/lib/facter/puppet_config_dir.rb]/ensure: defined content as '{md5}79477bc1f08beaf13d0a8438a1d2f1c2'
Notice: /File[/var/lib/puppet/lib/facter/puppet_settings.rb]/ensure: defined content as '{md5}9438c0839ae28dc52fffb8348ae5124f'
Notice: /File[/var/lib/puppet/lib/facter/raid.rb]/ensure: defined content as '{md5}f4ca7542cab0aed4b4b96289e28f69f0'
Notice: /File[/var/lib/puppet/lib/facter/root_home.rb]/ensure: defined content as '{md5}35702ae0c7410ec4d2101113e2f697fa'
Notice: /File[/var/lib/puppet/lib/facter/service_provider.rb]/ensure: defined content as '{md5}66cc42526eae631e306b397391f1f01c'
Notice: /File[/var/lib/puppet/lib/facter/util]/ensure: created
Notice: /File[/var/lib/puppet/lib/facter/util/puppet_settings.rb]/ensure: defined content as '{md5}9f1d2593d0ae56bfca89d4b9266aeee1'
Notice: /File[/var/lib/puppet/lib/hiera]/ensure: created
Notice: /File[/var/lib/puppet/lib/hiera/backend]/ensure: created
Notice: /File[/var/lib/puppet/lib/hiera/backend/httpyaml_backend.rb]/ensure: defined content as '{md5}0e9eef49702fcf899acc3f8e27248793'
Notice: /File[/var/lib/puppet/lib/hiera/backend/mwyaml_backend.rb]/ensure: defined content as '{md5}c073878c9658f5a1b40387a211da9c2c'
Notice: /File[/var/lib/puppet/lib/hiera/backend/nuyaml_backend.rb]/ensure: defined content as '{md5}72d99d3d289b2bea5b933b2355de7d55'
Notice: /File[/var/lib/puppet/lib/hiera/backend/proxy_backend.rb]/ensure: defined content as '{md5}36d8c8cc5956f88921acb0cfcd7f3556'
Notice: /File[/var/lib/puppet/lib/hiera/backend/puppetdb_backend.rb]/ensure: defined content as '{md5}7502879314b4c7775398a89c09a0c588'
Notice: /File[/var/lib/puppet/lib/hiera/backend/role_backend.rb]/ensure: defined content as '{md5}25cc4930fc7a6ba59dad69b164df4628'
Notice: /File[/var/lib/puppet/lib/hiera/httpcache.rb]/ensure: defined content as '{md5}577f41f0162b8f65e580eb451c0c6074'
Notice: /File[/var/lib/puppet/lib/hiera/mwcache.rb]/ensure: defined content as '{md5}b1688be31332837c5baba820dd28d134'
Notice: /File[/var/lib/puppet/lib/puppet]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/application]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/application/query.rb]/ensure: defined content as '{md5}6f9a79d593677f210fe754cedcf77fa2'
Notice: /File[/var/lib/puppet/lib/puppet/face]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/face/query.rb]/ensure: defined content as '{md5}d58cd9585586e9a98c3e78c6af3ab365'
Notice: /File[/var/lib/puppet/lib/puppet/functions]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/functions/deprecation.rb]/ensure: defined content as '{md5}59b803eaa15b5a559040497bffc172ae'
Notice: /File[/var/lib/puppet/lib/puppet/functions/is_a.rb]/ensure: defined content as '{md5}9dad7f8c9b75348cd97aca986ac0b29a'
Notice: /File[/var/lib/puppet/lib/puppet/functions/is_absolute_path.rb]/ensure: defined content as '{md5}96b217f26d06dbac87a2c6a8cfd2d8c8'
Notice: /File[/var/lib/puppet/lib/puppet/functions/is_array.rb]/ensure: defined content as '{md5}9292a646010d167417a1936b0b0c17b9'
Notice: /File[/var/lib/puppet/lib/puppet/functions/is_bool.rb]/ensure: defined content as '{md5}73957f9efd75ed8a7ab867f9de6da117'
Notice: /File[/var/lib/puppet/lib/puppet/functions/is_float.rb]/ensure: defined content as '{md5}af3bd6bb56878bac8cc4fe4f7564e4f9'
Notice: /File[/var/lib/puppet/lib/puppet/functions/is_ip_address.rb]/ensure: defined content as '{md5}ee231c66c3e039778bf46702d89815a6'
Notice: /File[/var/lib/puppet/lib/puppet/functions/is_ipv4_address.rb]/ensure: defined content as '{md5}900d33249906c4daa02aa79cac896548'
Notice: /File[/var/lib/puppet/lib/puppet/functions/is_ipv6_address.rb]/ensure: defined content as '{md5}568fba9af6a83c8b536fafcda82eb448'
Notice: /File[/var/lib/puppet/lib/puppet/functions/is_numeric.rb]/ensure: defined content as '{md5}33051800b886cc3b2119826b77c9821a'
Notice: /File[/var/lib/puppet/lib/puppet/functions/is_string.rb]/ensure: defined content as '{md5}230e9eabc5c9e1d8d5fb7b3c6c12b300'
Notice: /File[/var/lib/puppet/lib/puppet/functions/role.rb]/ensure: defined content as '{md5}369e35b14bf533791d4be3bf83ee6b05'
Notice: /File[/var/lib/puppet/lib/puppet/functions/type_of.rb]/ensure: defined content as '{md5}bec00841aae556993c926ab298bc81cd'
Notice: /File[/var/lib/puppet/lib/puppet/functions/validate_absolute_path.rb]/ensure: defined content as '{md5}54a610baa115c7505f1b35976b632a8e'
Notice: /File[/var/lib/puppet/lib/puppet/functions/validate_array.rb]/ensure: defined content as '{md5}9052b0026da174636c276a2512cf5acc'
Notice: /File[/var/lib/puppet/lib/puppet/functions/validate_bool.rb]/ensure: defined content as '{md5}fe979e402a5a3a19d013ce84b39ef06a'
Notice: /File[/var/lib/puppet/lib/puppet/functions/validate_hash.rb]/ensure: defined content as '{md5}92ea8fc21bbbf6cc41f6bb9cfcaefce7'
Notice: /File[/var/lib/puppet/lib/puppet/functions/validate_integer.rb]/ensure: defined content as '{md5}b0982b68a599262da2c6f2e032bc7713'
Notice: /File[/var/lib/puppet/lib/puppet/functions/validate_ip_address.rb]/ensure: defined content as '{md5}65a12af9a2c2a9c70d820d04d19ec891'
Notice: /File[/var/lib/puppet/lib/puppet/functions/validate_ipv4_address.rb]/ensure: defined content as '{md5}4a5039b99ac97cc0447faa343b9f7416'
Notice: /File[/var/lib/puppet/lib/puppet/functions/validate_ipv6_address.rb]/ensure: defined content as '{md5}fbdf685432416505fed27d5647c26f9c'
Notice: /File[/var/lib/puppet/lib/puppet/functions/validate_legacy.rb]/ensure: defined content as '{md5}d9f115f30c511cef536a821b94826094'
Notice: /File[/var/lib/puppet/lib/puppet/functions/validate_numeric.rb]/ensure: defined content as '{md5}41b2cc7335395f617c2bfbeac8f579da'
Notice: /File[/var/lib/puppet/lib/puppet/functions/validate_re.rb]/ensure: defined content as '{md5}42092f592ebf89b8a504b10c900230d8'
Notice: /File[/var/lib/puppet/lib/puppet/functions/validate_slength.rb]/ensure: defined content as '{md5}3ae6fcc3f60032c923d06ab3e457b84e'
Notice: /File[/var/lib/puppet/lib/puppet/functions/validate_string.rb]/ensure: defined content as '{md5}cc967a9d0ea156b2208d1760d7f6e1b2'
Notice: /File[/var/lib/puppet/lib/puppet/parser]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/abs.rb]/ensure: defined content as '{md5}32161bd0435fdfc2aec2fc559d2b454b'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/add_all_users.rb]/ensure: defined content as '{md5}dcc47e287dd5b9a33779e3cb1fa8e41c'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/any2array.rb]/ensure: defined content as '{md5}a81e71d6b67a551d38770ba9a1948a75'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/any2bool.rb]/ensure: defined content as '{md5}b5badfb4f9300d556b5b561d4ce599c5'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/apply_format.rb]/ensure: defined content as '{md5}12a1acc4133838f4ffdc63911494b9f5'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/array_concat.rb]/ensure: defined content as '{md5}e6775f4cde3798f96da97d5481b9d544'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/assert_private.rb]/ensure: defined content as '{md5}1365284f9e474ecec24cfe43ee8e7cf4'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/base64.rb]/ensure: defined content as '{md5}ac8d0565df2931470447effa502c9ad3'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/basename.rb]/ensure: defined content as '{md5}c61952b3f68fd86408c84fca2c3febb1'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/bool2num.rb]/ensure: defined content as '{md5}f953f5fc094c2ae3908a72d8840ba291'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/bool2str.rb]/ensure: defined content as '{md5}6334ac6d24a8aa49a2243fb425f47311'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/camelcase.rb]/ensure: defined content as '{md5}71c67b71eac4b7f46a0dd22cb915d2e6'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/capitalize.rb]/ensure: defined content as '{md5}da131748a9d32da9eb0b6438e39377eb'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/ceiling.rb]/ensure: defined content as '{md5}dfa9b1c75ce89344026b3b5aed2d190f'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/chomp.rb]/ensure: defined content as '{md5}2b7dc42f9967edd34cfa0ba9a97229ca'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/chop.rb]/ensure: defined content as '{md5}0ec76f54afd94201f35785dfeb2092b5'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/clamp.rb]/ensure: defined content as '{md5}d19b8dfc573ed2eff7e3536ae1e5d596'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/compile_redirects.rb]/ensure: defined content as '{md5}a8f1398c8fda61b1293c38c5a3a8f945'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/concat.rb]/ensure: defined content as '{md5}a1b0233e58f1f8095a74fe9300c74a9b'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/configparser_format.rb]/ensure: defined content as '{md5}bf20c790d6580038372c0f957fb780cf'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/conflicts.rb]/ensure: defined content as '{md5}91f9b3b37dcc1f553f76978306fe9a53'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/conftool.rb]/ensure: defined content as '{md5}20a23a832476b328ce469a367ed6fde6'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/convert_base.rb]/ensure: defined content as '{md5}c3b3e59a49318af98dcb88aed7156629'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/count.rb]/ensure: defined content as '{md5}9eb74eccd93e2b3c87fd5ea14e329eba'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/cron_splay.rb]/ensure: defined content as '{md5}72a046df5030fd87e1d06752e43886e9'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/deep_merge.rb]/ensure: defined content as '{md5}d83696855578fb81b64b9e92b9c7cc7c'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/defined_with_params.rb]/ensure: defined content as '{md5}80eed5cedb1b5134e6a1cf44e86ae60a'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/delete.rb]/ensure: defined content as '{md5}873d8244e7ecf7b2e56a46c2b3e09b4e'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/delete_at.rb]/ensure: defined content as '{md5}6bc24b79390d463d8be95396c963381a'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/delete_regex.rb]/ensure: defined content as '{md5}2fd4485e2526510dfde0a6687e23ee6e'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/delete_undef_values.rb]/ensure: defined content as '{md5}b32d4a3925753b2eb2c318cbd7f14404'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/delete_values.rb]/ensure: defined content as '{md5}39b147f7d369bb5f809044b6341954a2'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/deprecation.rb]/ensure: defined content as '{md5}c9c5dc3f48fdb7e20d02c18d0599f697'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/difference.rb]/ensure: defined content as '{md5}e31b95fbaf974cf853a510177368bfb9'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/dig.rb]/ensure: defined content as '{md5}1a2a8918f646c13dcb9876a22f9295ab'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/dig44.rb]/ensure: defined content as '{md5}8f4255bd98fe6a558cdb43a851e58ad3'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/dirname.rb]/ensure: defined content as '{md5}8a5579f9a9a13fd737ba65eccf8e6d5a'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/dos2unix.rb]/ensure: defined content as '{md5}be8359a5106a7832be4180e8207dd586'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/downcase.rb]/ensure: defined content as '{md5}73121616d73339cf8dd10e0de61a6c50'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/empty.rb]/ensure: defined content as '{md5}b4ad0c3c00cbc56f745fbc05af1efa00'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/enclose_ipv6.rb]/ensure: defined content as '{md5}581bc163291824909d1700909db96512'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/ensure_directory.rb]/ensure: defined content as '{md5}15f703b66860060cf4f9c0e25dc1bd0a'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/ensure_link.rb]/ensure: defined content as '{md5}38dd1a7e22c3681619e392c9f26dedf0'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/ensure_mounted.rb]/ensure: defined content as '{md5}9e13cef804a1472597b67ee5018c2bb1'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/ensure_packages.rb]/ensure: defined content as '{md5}952334ddf12d9e10b1c1658e63dc662d'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/ensure_resource.rb]/ensure: defined content as '{md5}de703fe63392b939fc2b4392975263de'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/ensure_resources.rb]/ensure: defined content as '{md5}c92d8b69d6354eda24aa3a13d88177b2'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/ensure_service.rb]/ensure: defined content as '{md5}b271d5c104b84a40f8019f9a5aae07f7'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/flatten.rb]/ensure: defined content as '{md5}25777b76f9719162a8bab640e5595b7a'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/floor.rb]/ensure: defined content as '{md5}42cad4c689231a51526c55a6f0985d1f'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/fqdn_rand_string.rb]/ensure: defined content as '{md5}9ac5f18e563094aee62ef7586267025d'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/fqdn_rotate.rb]/ensure: defined content as '{md5}770d510a2e50d19b2dd42b6edef3fb1f'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/fqdn_uuid.rb]/ensure: defined content as '{md5}d357e8837ba2ed8196e926f3697be521'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/get_clusters.rb]/ensure: defined content as '{md5}4f6c78d45b9fceaa5252ca248cd0df0a'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/get_module_path.rb]/ensure: defined content as '{md5}d4bf50da25c0b98d26b75354fa1bcc45'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/getparam.rb]/ensure: defined content as '{md5}440a9c381b9ad589504e2e9919e83c06'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/getvar.rb]/ensure: defined content as '{md5}0c8c5cef7e158e232a8cf6e42c10d0ff'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/grep.rb]/ensure: defined content as '{md5}5682995af458b05f3b53dd794c4bf896'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/has_interface_with.rb]/ensure: defined content as '{md5}e135f09dbecc038c3aa9ae03127617ef'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/has_ip_address.rb]/ensure: defined content as '{md5}ee207f47906455a5aa49c4fb219dd325'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/has_ip_network.rb]/ensure: defined content as '{md5}b4d726c8b2a0afac81ced8a3a28aa731'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/has_key.rb]/ensure: defined content as '{md5}7cd9728c38f0b0065f832dabd62b0e7e'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/hash.rb]/ensure: defined content as '{md5}784f8d2d114eb4c9414ebd8a6868f82f'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/hash_deselect_re.rb]/ensure: defined content as '{md5}c2c1bb824d1a791962730139e5074696'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/hash_select_re.rb]/ensure: defined content as '{md5}a2bc7680a971bab73b695b3b285ca69d'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/htpasswd.rb]/ensure: defined content as '{md5}ef64aa0291d2018b0b435795d00ac33d'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/ini.rb]/ensure: defined content as '{md5}8511674adcbcaf07e13e4f54e0769e7d'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/init_template.rb]/ensure: defined content as '{md5}143b173d73719f8103cbc55a87e90bc5'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/intersection.rb]/ensure: defined content as '{md5}c8f4f8b861c9c297c87b08bdbfb94caa'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/ipresolve.rb]/ensure: defined content as '{md5}b1af0cfdc5ddeaa20e1a6f6fe72984ea'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/is_absolute_path.rb]/ensure: defined content as '{md5}1ce9a6d1cd0a79087d73cb879ed04542'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/is_array.rb]/ensure: defined content as '{md5}343cdcafdb6d353773f14e6d8864d915'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/is_bool.rb]/ensure: defined content as '{md5}92589666ef0b0cccafb2c80bb786b3ff'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/is_domain_name.rb]/ensure: defined content as '{md5}6ca1f2708add756a6803b29d593d5830'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/is_email_address.rb]/ensure: defined content as '{md5}82573a431edf5a0bf9847393af27566a'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/is_float.rb]/ensure: defined content as '{md5}363e152a18e18997be12ab61b56c3160'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/is_function_available.rb]/ensure: defined content as '{md5}628428bbcd9313ce09783d9484330e09'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/is_hash.rb]/ensure: defined content as '{md5}8c7d9a05084dab0389d1b779c8a05b1a'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/is_integer.rb]/ensure: defined content as '{md5}c5f4aeb52da82e1d87f36b4b34349767'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/is_ip_address.rb]/ensure: defined content as '{md5}87316b88db72b2359351c7ab4426cdb4'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/is_ipv4_address.rb]/ensure: defined content as '{md5}0e2862f6280bff0f4c189b9f5f37a420'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/is_ipv6_address.rb]/ensure: defined content as '{md5}066706256a142573a96afb4615fe99a6'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/is_mac_address.rb]/ensure: defined content as '{md5}6dd3c96437d49e68630869b0b464e7f2'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/is_numeric.rb]/ensure: defined content as '{md5}72d25c5f9daae0220d64636efc6977b6'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/is_string.rb]/ensure: defined content as '{md5}028bac4d729e7b9990fbd5afb8cbc21b'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/join.rb]/ensure: defined content as '{md5}a285a05c015ae278608f6454aef211ea'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/join_keys_to_values.rb]/ensure: defined content as '{md5}c17ad05bca6eb1a409d355f3f7637115'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/kafka_cluster_name.rb]/ensure: defined content as '{md5}582a0ab4eda21993989a0a2dde626f13'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/kafka_config.rb]/ensure: defined content as '{md5}07a69c8cbcf02e5f8b78ca2bfc698dd1'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/keys.rb]/ensure: defined content as '{md5}eb6ac815ea14fbf423580ed903ef7bad'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/load_module_metadata.rb]/ensure: defined content as '{md5}805c5476a6e7083d133e167129885924'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/loadjson.rb]/ensure: defined content as '{md5}ffecf61ba2ec8011915d37009b1a273e'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/loadyaml.rb]/ensure: defined content as '{md5}668265f14327cba1d1400f2078b7b26b'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/lstrip.rb]/ensure: defined content as '{md5}20a9b1fa077c16f34e0ef5448b895698'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/max.rb]/ensure: defined content as '{md5}f652fd0b46ef7d2fbdb42b141f8fdd1d'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/member.rb]/ensure: defined content as '{md5}2b5d7fb8f87f1c7d195933c57ca32e91'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/merge.rb]/ensure: defined content as '{md5}f3dcc5c83440cdda2036cce69b61a14b'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/merge_config.rb]/ensure: defined content as '{md5}c6b05aab88549e5d9639f787539d1939'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/min.rb]/ensure: defined content as '{md5}0d2a1b7e735ab251c5469e735fa3f4c6'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/mount_nfs_volume.rb]/ensure: defined content as '{md5}0d9c104e333ab1a4dffb3244e2deaf2f'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/mysql_password.rb]/ensure: defined content as '{md5}3c375e08f7372795ad25d51d43995f96'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/num2bool.rb]/ensure: defined content as '{md5}605c12fa518c87ed2c66ae153e0686ce'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/nutcracker_pools.rb]/ensure: defined content as '{md5}b6b98ba8e7b5257c2306d699aa8b5be2'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/ordered_json.rb]/ensure: defined content as '{md5}2f351f5d5e3cbbc13b062c82bcae8de1'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/ordered_yaml.rb]/ensure: defined content as '{md5}25400137e3fe5d25388fd53240e6e68b'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/os_version.rb]/ensure: defined content as '{md5}c90cf7334e7423ab2b75089bdcee7093'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/parsejson.rb]/ensure: defined content as '{md5}15165fd3807d9f3d657697fa854d643d'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/parseyaml.rb]/ensure: defined content as '{md5}db54578d9d798ced75952217cf11b737'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/pdbfactquery.rb]/ensure: defined content as '{md5}6bc06fa7c5fecca1f09e19b50a84ae37'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/pdbnodequery.rb]/ensure: defined content as '{md5}8c14e9cea5cf2623580dc588cf70d80f'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/pdbnodequery_all.rb]/ensure: defined content as '{md5}3af4db0df0672bdb2d2f2e676b60b2f2'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/pdbquery.rb]/ensure: defined content as '{md5}577276bb1ed8cc88460a702244a3c0fe'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/pdbresourcequery.rb]/ensure: defined content as '{md5}1a2040b64c1a7e4f6c6c5909b4b0f5e0'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/pdbresourcequery_all.rb]/ensure: defined content as '{md5}4555b8fed1e30051c17770bac2329465'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/pdbstatusquery.rb]/ensure: defined content as '{md5}c61372b06340a6fee1bf492989368cd2'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/php_ini.rb]/ensure: defined content as '{md5}5e8904528c557456d6223f3053349b1d'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/phpdump.rb]/ensure: defined content as '{md5}d0aa7ae7f2199d9a13ffa83db6aad713'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/pick.rb]/ensure: defined content as '{md5}bf01f13bbfe2318e7f6a302ac7c4433f'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/pick_default.rb]/ensure: defined content as '{md5}ad3ea60262de408767786d37a54d45dc'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/pick_initscript.rb]/ensure: defined content as '{md5}bf6656acf34a73597fea72a2419e7d7e'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/prefix.rb]/ensure: defined content as '{md5}e377fd64bd63dde6c9660aa75aca4942'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/private.rb]/ensure: defined content as '{md5}1500a21d5cf19961c5b1d476df892d92'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/pry.rb]/ensure: defined content as '{md5}79a48e45196e4bbf0a3e658781513cf4'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/puppet_ssldir.rb]/ensure: defined content as '{md5}1812770758dc26f99f43138853e361fe'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/pw_hash.rb]/ensure: defined content as '{md5}d82221f667050026cd6d155432a31802'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/query_facts.rb]/ensure: defined content as '{md5}7c9917e4a43d8cfea695bbf6b0889746'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/query_nodes.rb]/ensure: defined content as '{md5}4566604125199f885ce8bc23addb519f'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/query_resources.rb]/ensure: defined content as '{md5}a2b7ff9c144477b9e888d924ba986d3e'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/range.rb]/ensure: defined content as '{md5}2203172c91cc324e85433618597be3be'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/redis_get_instances.rb]/ensure: defined content as '{md5}136ab93948667ff2a94b8d1221691c37'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/redis_shard_hosts.rb]/ensure: defined content as '{md5}9b1e0a69bb5d833802e586e4bce9aeb8'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/regexpescape.rb]/ensure: defined content as '{md5}f9cfb10f6acd1e318bf60bb7f561bde3'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/reject.rb]/ensure: defined content as '{md5}689f6a7c961a55fe9dcd240921f4c7f9'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/require_package.rb]/ensure: defined content as '{md5}d94ecf9967d9af5033ddf25d822118b0'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/requires_os.rb]/ensure: defined content as '{md5}9d4e52c05e091747c401bd7bf21f81cf'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/requires_realm.rb]/ensure: defined content as '{md5}96e334fb2b41d65256b12f8c9f390c5f'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/reverse.rb]/ensure: defined content as '{md5}b234b54b8cd62b2d67ccd70489ffdccf'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/rstrip.rb]/ensure: defined content as '{md5}b4e4ada41f7c1d2fcad073ce6344980f'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/secret.rb]/ensure: defined content as '{md5}54f75ee3f39722d86b601266d18d3150'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/seeded_rand.rb]/ensure: defined content as '{md5}2ad22e7613d894ae779c0c5b0e65dade'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/shell_escape.rb]/ensure: defined content as '{md5}b9298edaf48883d7b7d2dd8c76c25d2f'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/shell_exports.rb]/ensure: defined content as '{md5}d88b058981020bcb0b8888a5f55cb6d6'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/shell_join.rb]/ensure: defined content as '{md5}60817e59ed6eda2cc536d15c88d0d280'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/shell_split.rb]/ensure: defined content as '{md5}ed608feaf0f1c36a54a3a9459384d414'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/shuffle.rb]/ensure: defined content as '{md5}d50f72b0aeb921e64d2482f62488e2f3'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/size.rb]/ensure: defined content as '{md5}ab3b5b8cf2369d76969a7cb2564e018f'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/slice_network_constants.rb]/ensure: defined content as '{md5}9bb415213817ccee14045ba1f6c2340a'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/sort.rb]/ensure: defined content as '{md5}504b033b438461ca4f9764feeb017833'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/squeeze.rb]/ensure: defined content as '{md5}541f85b4203b55c9931d3d6ecd5c75f8'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/ssl_ciphersuite.rb]/ensure: defined content as '{md5}94278899aaa1fc9e62076183795f8575'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/str2bool.rb]/ensure: defined content as '{md5}44b411fa76cd6713e37354280b102587'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/str2saltedsha512.rb]/ensure: defined content as '{md5}49afad7b386be38ce53deaefef326e85'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/strftime.rb]/ensure: defined content as '{md5}e02e01a598ca5d7d6eee0ba22440304a'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/strip.rb]/ensure: defined content as '{md5}85d70ab95492e3e4ca5f0b5ec3f284a9'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/suffix.rb]/ensure: defined content as '{md5}082519722ed6f237dd3a9a7e316ab2e4'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/swapcase.rb]/ensure: defined content as '{md5}b17a9f3cb0271451d309e4b4f52dd651'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/systemd_template.rb]/ensure: defined content as '{md5}5af8296860b31e9e3069e6508f91ece8'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/sysvinit_template.rb]/ensure: defined content as '{md5}0bf2e8ba0d1d5c0c810eea69ee183016'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/time.rb]/ensure: defined content as '{md5}8cb0b8320c60b4a21725634154a9f1db'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/to_bytes.rb]/ensure: defined content as '{md5}65437027687b6172173b3a211a799e37'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/to_milliseconds.rb]/ensure: defined content as '{md5}f96f2e5af3b2f22babf170a00375a961'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/to_seconds.rb]/ensure: defined content as '{md5}e3f5659d6510239e88662d6cb9c0003d'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/try_get_value.rb]/ensure: defined content as '{md5}09cd079ec5f0e5e2ac862c9ebe0f54fe'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/type.rb]/ensure: defined content as '{md5}4709f7ab8a8aad62d77a3c5d91a3aa08'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/type3x.rb]/ensure: defined content as '{md5}f9bf4de8341afb0c677c26b40ec8a2b2'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/union.rb]/ensure: defined content as '{md5}3cf57ea53f2522f586264feb67293cd6'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/unique.rb]/ensure: defined content as '{md5}c1bb4a8aeebd09ba3e4c8bc3702cfd60'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/unique_users.rb]/ensure: defined content as '{md5}1cdbc7faea922ffa23c234d7be17ab81'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/unix2dos.rb]/ensure: defined content as '{md5}b1f5087fcaca69d9395094204cce887a'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/upcase.rb]/ensure: defined content as '{md5}8decededec9eb33e58f961eb86f0888f'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/upstart_template.rb]/ensure: defined content as '{md5}6002c7ffc2137fd576f9da66e3cc4afd'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/uriescape.rb]/ensure: defined content as '{md5}d912ba09ba3f58c70988e662e05ffbe8'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_absolute_path.rb]/ensure: defined content as '{md5}c4b12e101055380386a235cd2c92ad10'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_array.rb]/ensure: defined content as '{md5}bee8327014714d4cd8dbb5c46611f594'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_array_re.rb]/ensure: defined content as '{md5}f564bb4f308075bb97bbd42494617395'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_augeas.rb]/ensure: defined content as '{md5}61e828e7759ba3e1e563e1fdd68aa80f'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_bool.rb]/ensure: defined content as '{md5}971700229c4b581f67f6fadc9019eb2c'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_cmd.rb]/ensure: defined content as '{md5}7df12370db442eddddcf4dd7a5364b5e'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_email_address.rb]/ensure: defined content as '{md5}a72656cbfeda622cdd5cfdafdf464095'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_ensure.rb]/ensure: defined content as '{md5}fabbbe98084b794bd35039838fee3fef'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_hash.rb]/ensure: defined content as '{md5}c2ae6299148ea200f8dfcf9204875182'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_integer.rb]/ensure: defined content as '{md5}65aa35f7450794aaadb6ad2c2e114df7'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_ip_address.rb]/ensure: defined content as '{md5}b23c3d5ce6839e32d0186411147a6a44'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_ipv4_address.rb]/ensure: defined content as '{md5}593e8f832469cb6a48c5f16ee66c3b2d'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_ipv6_address.rb]/ensure: defined content as '{md5}48d3733012818993eae662839183d139'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_numeric.rb]/ensure: defined content as '{md5}0e36d370262b8bdef2f88f0a3cb5b30e'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_re.rb]/ensure: defined content as '{md5}d5963c404e3ac1670553f306221c2655'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_slength.rb]/ensure: defined content as '{md5}6cbcfe15378ca4a780bac786223aacac'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_string.rb]/ensure: defined content as '{md5}8afa7b0dcfe17bfbbb5704ad54664cc2'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/validate_x509_rsa_key_pair.rb]/ensure: defined content as '{md5}f17427dfc42128dc1df0ab04f37942e5'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/values.rb]/ensure: defined content as '{md5}066a6e4170e5034edb9a80463dff2bb5'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/values_at.rb]/ensure: defined content as '{md5}325a899e0201e8df5bd483fec6f12d76'
Notice: /File[/var/lib/puppet/lib/puppet/parser/functions/zip.rb]/ensure: defined content as '{md5}a89d5e802bc1e63e52020c2ddbaaca2c'
Notice: /File[/var/lib/puppet/lib/puppet/provider]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/database]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/database/mysql.rb]/ensure: defined content as '{md5}5ca8a2fd43b8e9b8fd87779017b94486'
Notice: /File[/var/lib/puppet/lib/puppet/provider/database_grant]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/database_grant/mysql.rb]/ensure: defined content as '{md5}1836e5466a1425d03f853af6f2acbaa7'
Notice: /File[/var/lib/puppet/lib/puppet/provider/database_user]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/database_user/mysql.rb]/ensure: defined content as '{md5}daf038e7ce0adea94fdcb2536a7b13ea'
Notice: /File[/var/lib/puppet/lib/puppet/provider/file_line]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/file_line/ruby.rb]/ensure: defined content as '{md5}b3d2d66ac39c43a1e7a8dd6b78952d65'
Notice: /File[/var/lib/puppet/lib/puppet/provider/filesystem]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/filesystem/aix.rb]/ensure: defined content as '{md5}522f95e24a10299934d589e432e01a3f'
Notice: /File[/var/lib/puppet/lib/puppet/provider/filesystem/lvm.rb]/ensure: defined content as '{md5}eb5009786de0a841b206b6ece490bc06'
Notice: /File[/var/lib/puppet/lib/puppet/provider/gridengine_resource]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/gridengine_resource/parsed.rb]/ensure: defined content as '{md5}6f4a5aa2dfc498746936f2e63fba390b'
Notice: /File[/var/lib/puppet/lib/puppet/provider/logical_volume]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/logical_volume/aix.rb]/ensure: defined content as '{md5}86ca794d7fbb5c9932d19de5e95569b9'
Notice: /File[/var/lib/puppet/lib/puppet/provider/logical_volume/lvm.rb]/ensure: defined content as '{md5}c96c057adcef80c079e0702f3f599f54'
Notice: /File[/var/lib/puppet/lib/puppet/provider/package]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/package/docker.rb]/ensure: defined content as '{md5}66af88488ecc4a9e7ae77a2b807daac6'
Notice: /File[/var/lib/puppet/lib/puppet/provider/package/scap3.rb]/ensure: defined content as '{md5}e20e950d44dfe97831feb8d8aebb4e75'
Notice: /File[/var/lib/puppet/lib/puppet/provider/physical_volume]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/physical_volume/aix.rb]/ensure: defined content as '{md5}9ece0f988d380fbdb10421d78cac99fd'
Notice: /File[/var/lib/puppet/lib/puppet/provider/physical_volume/lvm.rb]/ensure: defined content as '{md5}3a6d71760fe968947336bde268eb17b4'
Notice: /File[/var/lib/puppet/lib/puppet/provider/scap_source]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/scap_source/default.rb]/ensure: defined content as '{md5}76290feb6e7f2f592cf09914b6ee6d30'
Notice: /File[/var/lib/puppet/lib/puppet/provider/volume_group]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/provider/volume_group/aix.rb]/ensure: defined content as '{md5}997bde52fcf9d8d45d67dfc9d9dd18ee'
Notice: /File[/var/lib/puppet/lib/puppet/provider/volume_group/lvm.rb]/ensure: defined content as '{md5}a1a4bbcf8e15f18429588ecf3fc4237e'
Notice: /File[/var/lib/puppet/lib/puppet/reports]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/reports/logstash.rb]/ensure: defined content as '{md5}bd810681eaa70faa43cbda1c6756e8e4'
Notice: /File[/var/lib/puppet/lib/puppet/reports/servermon.rb]/ensure: defined content as '{md5}4d7e0065515dae2f79d81b783d51f34c'
Notice: /File[/var/lib/puppet/lib/puppet/reports/statsd.rb]/ensure: defined content as '{md5}4ce268e9047fa1b72d1f55d4eb3e0683'
Notice: /File[/var/lib/puppet/lib/puppet/type]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppet/type/anchor.rb]/ensure: defined content as '{md5}bbd36bb49c3b554f8602d8d3df366c0c'
Notice: /File[/var/lib/puppet/lib/puppet/type/database.rb]/ensure: defined content as '{md5}a5e9e5edb5aa67bbddd17fb9096c9eae'
Notice: /File[/var/lib/puppet/lib/puppet/type/database_grant.rb]/ensure: defined content as '{md5}8a9d41da37ab8450f656b0e50c0f3c1e'
Notice: /File[/var/lib/puppet/lib/puppet/type/database_user.rb]/ensure: defined content as '{md5}3c1569f97f0b2c76437503e3d47d520b'
Notice: /File[/var/lib/puppet/lib/puppet/type/etcd_role.rb]/ensure: defined content as '{md5}1b1296c10ef48be5fc6598e53c0896ce'
Notice: /File[/var/lib/puppet/lib/puppet/type/etcd_user.rb]/ensure: defined content as '{md5}8ff12766a0e86201efc7ff0990eb9820'
Notice: /File[/var/lib/puppet/lib/puppet/type/file_line.rb]/ensure: defined content as '{md5}34bfa8c554faea5e9ae1a6c519824de1'
Notice: /File[/var/lib/puppet/lib/puppet/type/filesystem.rb]/ensure: defined content as '{md5}b244e53c2d16bfb202b62794f957f4ab'
Notice: /File[/var/lib/puppet/lib/puppet/type/gridengine_resource.rb]/ensure: defined content as '{md5}84dec23edaa11f72ef4df03d78b0e745'
Notice: /File[/var/lib/puppet/lib/puppet/type/logical_volume.rb]/ensure: defined content as '{md5}8071b87ffa42a808ffeaba32d18cf703'
Notice: /File[/var/lib/puppet/lib/puppet/type/physical_volume.rb]/ensure: defined content as '{md5}4f917fbbb3dc8ceac927e661f9dd913a'
Notice: /File[/var/lib/puppet/lib/puppet/type/scap_source.rb]/ensure: defined content as '{md5}ca037233b109c63e62d92b57787d7cf2'
Notice: /File[/var/lib/puppet/lib/puppet/type/volume_group.rb]/ensure: defined content as '{md5}55d3b8084c7b1631bb7aa64dca4b057e'
Notice: /File[/var/lib/puppet/lib/puppetdb]/ensure: created
Notice: /File[/var/lib/puppet/lib/puppetdb.rb]/ensure: defined content as '{md5}6f1592f0369d4ef23a28e58fc61e5419'
Notice: /File[/var/lib/puppet/lib/puppetdb/astnode.rb]/ensure: defined content as '{md5}736c8851f7ed9f47f9dc0c9727388154'
Notice: /File[/var/lib/puppet/lib/puppetdb/connection.rb]/ensure: defined content as '{md5}a22914dc6d4611952e47afa53ffc48e4'
Notice: /File[/var/lib/puppet/lib/puppetdb/grammar.racc]/ensure: defined content as '{md5}76cd28f9ea5b879aeb1c2039683df24d'
Notice: /File[/var/lib/puppet/lib/puppetdb/lexer.rb]/ensure: defined content as '{md5}6c29d8912fd05729a5166d142c91b79e'
Notice: /File[/var/lib/puppet/lib/puppetdb/lexer.rex]/ensure: defined content as '{md5}e8a7722365726dcab5ed8489f4130477'
Notice: /File[/var/lib/puppet/lib/puppetdb/parser.rb]/ensure: defined content as '{md5}20428aa77fb78f35f814e4c582607587'
Notice: /File[/var/lib/puppet/lib/puppetdb/util.rb]/ensure: defined content as '{md5}5618f606db5a476978ba188ba82503db'
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Badly-formed puppet certname: localhost.codfw.labtest at /etc/puppet/manifests/realm.pp:31:9 on node localhost.codfw.labtest
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

A second run:

root@localhost:~# puppet agent -t -v
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get certificate CRL for /CN=puppet]
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get certificate CRL for /CN=puppet]
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get certificate CRL for /CN=puppet]
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get certificate CRL for /CN=puppet]
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get certificate CRL for /CN=puppet]
Info: Loading facts
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get certificate CRL for /CN=puppet]
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get certificate CRL for /CN=puppet]

I keep working on this.

By copying the /var/lib/puppet/ssl/crl.pem file from a VM in main I can run puppet. Now the issue is the catalog itself. The VM in labtest has no name, so there is no catalog for it:

root@localhost:~# puppet agent -t -v
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Badly-formed puppet certname: localhost.codfw.labtest at /etc/puppet/manifests/realm.pp:31:9 on node localhost.codfw.labtest
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

I will try to persist changes until here in the base image.

Change 428402 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Labtest: try to standardize on a service name for the puppetmaster

https://gerrit.wikimedia.org/r/428402

Change 428402 merged by Andrew Bogott:
[operations/puppet@production] Labtest: try to standardize on a service name for the puppetmaster

https://gerrit.wikimedia.org/r/428402

Change 428416 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labtestpuppetmaster: define 'ca_server'

https://gerrit.wikimedia.org/r/428416

Change 428416 merged by Andrew Bogott:
[operations/puppet@production] labtestpuppetmaster: define 'ca_server'

https://gerrit.wikimedia.org/r/428416

Change 428425 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] labtestpuppetmaster: further attempts to set the ca_server

https://gerrit.wikimedia.org/r/428425

Change 428425 merged by Andrew Bogott:
[operations/puppet@production] labtestpuppetmaster: further attempts to set the ca_server

https://gerrit.wikimedia.org/r/428425

Andrew mentioned this in T192848: puppetmaster puppet.conf refers to noexistent files.Apr 23 2018, 8:29 PM

I organized the puppet code a bit and then rebuilt everything on the master:

Fresh start:

dpkg --purge puppetmaster puppetmaster-passenger

rm -rf /var/lib/puppet/server

rm -rf /var/lib/puppet/server_data

rm -rf /var/lib/puppet/ssl

puppet cert clean labtestpuppetmaster2001.wikimedia.org on puppetmaster1001

Setup:

puppet agent -tv

(sign cert on puppetmaster1001)

puppet agent -tv

puppet cert generate labtestpuppetmaster2001.wikimedia.org

Committed the following files to the private puppet repo:

/var/lib/puppet/server/ssl/certs/labtest-puppetmaster.wikimedia.org.pem as /srv/private/modules/secret/secrets/puppetmaster/labtest-puppetmaster.wikimedia.org_pubkey.pem

/var/lib/puppet/server/ssl/private_keys/labtest-puppetmaster.wikimedia.org.pem as /srv/private/modules/secret/secrets/puppetmaster/labtest-puppetmaster.wikimedia.org_privkey.pem

After all that, the behavior for a new VM is exactly the same. The client requests a cert, I sign it on the master, and then "unable to get local issuer certificate for /CN=puppet"

In T181523#4151521, @Andrew wrote:

/var/lib/puppet/server/ssl/certs/labtest-puppetmaster.wikimedia.org.pem as /srv/private/modules/secret/secrets/puppetmaster/labtest-puppetmaster.wikimedia.org_pubkey.pem

/var/lib/puppet/server/ssl/private_keys/labtest-puppetmaster.wikimedia.org.pem as /srv/private/modules/secret/secrets/puppetmaster/labtest-puppetmaster.wikimedia.org_privkey.pem

That certificate was signed by an unknown CA:

X509v3 Authority Key Identifier: 
    keyid:2B:38:8B:93:8C:7E:36:BE:EE:2E:31:5F:77:E7:F1:19:54:FA:AA:AC

labpuppetmaster certificate was signed by this CA:

X509v3 Authority Key Identifier: 
    keyid:3C:98:64:31:78:C7:D8:0F:6F:FB:AA:15:6C:7D:63:3A:68:B4:DE:A9

Both have Issuer: CN=Puppet CA: puppet, but the first one is not included in the VMs.

We could be doing something wrong. From an IRC conversation with @jcrespo @akosiaris and @Vgutierrez we reached these conclusions:

labtestpuppetmaster should use a self-signed certificate to deal with his own clients
labtestpuppetmaster should use a certificate signed by puppetmaster1001 only for the agent operations
VMs don't need to have any CA beforehand, they blindly accept what comes from the puppetmaster (labtestpuppetmaster) in the first run

Things we are doing wrong:

the server certificate in use by labtestpuppetmaster (apache) is the puppet agent certificate, issued by puppetmaster1001. Instead, this should be a self-signed certificate / root CA.
VMs should blindy accept what comes from labtestpuppetmaster on the first run. This only happens if /var/lib/puppet/ssl is empty, and we currently build our images with this not empty?

I still have no definitive answers on why labpuppetmater VMs work but labtestpuppetmaster VMs don't.

herron added a subscriber: herron.Apr 24 2018, 1:43 PM

VMs should blindy accept what comes from labtestpuppetmaster on the first run. This only happens if /var/lib/puppet/ssl is empty, and we currently build our images with this not empty?

In the first boot script, new VMs do this before enabling puppet:

# Make sure nothing has leaked in certwise
rm -rf /var/lib/puppet/ssl/*

Just digging in to this now, and this may already be known, so apologies in advance if this is duplicate info.

In the description I noticed

[agent]
server = labtestpuppetmaster2001.wikimedia.org

But it looks like labtestpuppetmaster2001.wikimedia.org is not a name on the cert served by labtestpuppetmaster2001. Instead it has:

# openssl s_client -connect labtestpuppetmaster2001.wikimedia.org:8140 | openssl x509 -text | grep CN
depth=1 CN = Puppet CA: puppet
verify error:num=19:self signed certificate in certificate chain
        Issuer: CN = Puppet CA: puppet
        Subject: CN = labtest-puppetmaster.wikimedia.org

Depending on the server name used I was able to produce both a failed run with "unable to get local issuer certificate" and a successful run.

First, move away /var/lib/puppet/ssl so a new cert is created and try a puppet run against labtestpuppetmaster2001

root@mx-keith-stretch1:~# mv /var/lib/puppet/ssl /var/lib/puppet/ssl.away
root@mx-keith-stretch1:~# puppet agent -t --server labtestpuppetmaster2001.wikimedia.org
Info: Creating a new SSL key for mx-keith-stretch1.project-smtp.eqiad.wmflabs
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for mx-keith-stretch1.project-smtp.eqiad.wmflabs
Info: Certificate Request fingerprint (SHA256): F8:79:C9:C5:D7:7A:ED:A3:8B:BD:C2:38:9E:43:7D:9F:FE:69:99:C2:98:A7:9E:85:18:B5:6F:3F:F0:90:7C:8B
Info: Caching certificate for mx-keith-stretch1.project-smtp.eqiad.wmflabs
Error: Could not request certificate: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=labtestpuppetmaster2001.wikimedia.org]
Exiting; failed to retrieve certificate and waitforcert is disabled

Ok, a new cert was created but this errors out with unable to get local issuer certificate for /CN=labtestpuppetmaster2001.wikimedia.org

Confirmed the cert was signed on labtestpuppetmaster2001 and tried a run against labtestpuppetmaster2001.wikimedia.org just for kicks

root@mx-keith-stretch1:~# puppet agent -t --server labtestpuppetmaster2001.wikimedia.org
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=labtestpuppetmaster2001.wikimedia.org]
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=labtestpuppetmaster2001.wikimedia.org]
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=labtestpuppetmaster2001.wikimedia.org]
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=labtestpuppetmaster2001.wikimedia.org]
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=labtestpuppetmaster2001.wikimedia.org]
Info: Loading facts

It errors out again with unable to get local issuer certificate

Then I tried running with server name labtest-puppetmaster.wikimedia.org and unable to get local issuer certificate goes away.

root@mx-keith-stretch1:~# puppet agent -t --server labtest-puppetmaster.wikimedia.org
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts

Could it be fixed by setting server = labtest-puppetmaster.wikimedia.org on the test agents?

Change 428694 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] labs_bootstrapvz: add labtest-jessie specific firstboot.sh script

https://gerrit.wikimedia.org/r/428694

fgiunchedi added a subscriber: fgiunchedi.Apr 25 2018, 7:57 AM

We seem to have a working image:

Name: debian jessie (aborrero test 3) 
ID: b64c3d2a-06a3-4160-9012-afd1748cbd02

I will clean up other jessie images from labtest.

Change 428694 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] labs_bootstrapvz: address labtest issues

https://gerrit.wikimedia.org/r/428694

Change 429211 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] labs_bootstrapvz: firstboot.sh: bring back some resolv.conf magic

https://gerrit.wikimedia.org/r/429211

Last updates:

in /etc/puppet/puppet.conf, the server = puppet setting doesn't work (bad cert). If we use server = labtest-puppetmaster.wikimedia.org it does work. This is probably because we use the puppet agent cert (client) as the master/apache cert in labtestpuppetmaster.

we need some LDAP tunning, otherwise:

2018-04-26T16:35:50.917064+00:00 test20 nslcd[1241]: [8b4567] <group/member="puppet"> failed to bind to LDAP server ldap://ldap-labs.eqiad.wikimedia.org:389: Can't contact LDAP server
2018-04-26T16:35:50.917468+00:00 test20 nslcd[1241]: [8b4567] <group/member="puppet"> ldap_start_tls_s() failed (uri=ldap://ldap-labs.codfw.wikimedia.org:389): Can't contact LDAP server
2018-04-26T16:35:50.917876+00:00 test20 nslcd[1241]: [8b4567] <group/member="puppet"> failed to bind to LDAP server ldap://ldap-labs.codfw.wikimedia.org:389: Can't contact LDAP server
2018-04-26T16:35:50.918276+00:00 test20 nslcd[1241]: [8b4567] <group/member="puppet"> no available LDAP server found, sleeping 1 seconds

we need to properly adjust fqdn/hostname/domain/project name, otherwise the VM may request the wrong catalog:

puppet-agent[1751]: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Badly-formed puppet certname: test18.codfw.labtest at /etc/puppet/manifests/realm.pp:31:9 on node test18.codfw.labtest

not really a complete issue, but we need to proxy apt operations, otherwise apt-get update may have long timeout trying to fetch data over IPv6 which is unavailable

According to @Andrew, our goal is to have the simplest possible image, so we should try to fix some of those not from the VM but from server side.

Hey @fgiunchedi could you give us some help with the certificate issue?

it seems we are using the labtestpuppetmaster puppet clients certs to server apache/puppetmaster clients (VMs in the labtest deployment)
this cert above is generated by puppetmaster1001.eqiad.wmnet and is commited to the private repo for using in the puppetmaster::ca_server puppet class (which is used in labtestpuppetmaster given the role as puppet master / CA server)
@akosiaris suggested that this apache/puppetmaster cert should be a self-signed cert (aka root CA) which is generated at puppet package install time
there seems to be some chicken/egg problem. If we need the self-signed certificate auto-generated at package install time, but we need to commit that cert to the private repo, how is that supposed to work?

We will really appreciate any hint, docs or clarification :-) thanks!

In T181523#4162037, @aborrero wrote:

Hey @fgiunchedi could you give us some help with the certificate issue?

it seems we are using the labtestpuppetmaster puppet clients certs to server apache/puppetmaster clients (VMs in the labtest deployment)

this cert above is generated by puppetmaster1001.eqiad.wmnet and is commited to the private repo for using in the puppetmaster::ca_server puppet class (which is used in labtestpuppetmaster given the role as puppet master / CA server)

@akosiaris suggested that this apache/puppetmaster cert should be a self-signed cert (aka root CA) which is generated at puppet package install time

there seems to be some chicken/egg problem. If we need the self-signed certificate auto-generated at package install time, but we need to commit that cert to the private repo, how is that supposed to work?

We will really appreciate any hint, docs or clarification :-) thanks!

For sure! I'll explain how this works in production so it might shed some light on how it would work in labtest:

The puppetmaster has a vhost for puppet which agents talk to, this vhost uses certificates signed by the production puppet CA and of course has a SAN of puppet.
The private/public keypair for said certificate live in private.git (which is synchronized among all masters)
When provisioning a new puppet master the keypair is installed by puppet agent in the path expected by apache
puppet agent's ca_server setting is used only during CA requests, IOW when the agent needs to get its cert signed by the CA to be able to talk to the master.

The ca_server of course needs a CA to be able to operate, I take it labtest will be using a separate CA for this task?
I'm asking because:

@akosiaris suggested that this apache/puppetmaster cert should be a self-signed cert (aka root CA) which is generated at puppet package install time

Apache will need a certificate signed by the puppet CA, but not a self-signed certificate (which is different than a root CA)

Hope that helps!

Unfortunately this whole thing is complicated because there's different moving parts:

puppet master, which in some cases acts as a CA server
apache which talks tls to puppet agents and forwards the dn/subject presented by the agent to the puppet master

Change 429428 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmcs puppetmasters: make cert_secret_path configurable per deploy

https://gerrit.wikimedia.org/r/429428

Change 429428 merged by Andrew Bogott:
[operations/puppet@production] wmcs puppetmasters: make cert_secret_path configurable per deploy

https://gerrit.wikimedia.org/r/429428

Change 429437 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] bootstrapvz first boot: Go back to using named 'puppet' puppetmaster name

https://gerrit.wikimedia.org/r/429437

Change 429437 merged by Andrew Bogott:
[operations/puppet@production] bootstrapvz first boot: Go back to using named 'puppet' puppetmaster name

https://gerrit.wikimedia.org/r/429437

There were a lot of problems to hammer out here, but thanks to @aborrero's diagnosis things are running cleanly now. The only thing that doesn't work is switching over the puppetmaster from 'puppet' to 'labtest-puppetmaster.wikimedia.org' after the first puppet run. I've worked around that by just leaving it set to 'puppet' for good on labtest. I'm not sure if this is a bug or a feature... we might want to just do the same on main.

bd808 moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.Jun 4 2018, 12:01 AM

aborrero closed this task as Resolved.Jun 6 2018, 2:31 PM

Liuxinyu970226 removed a subscriber: Liuxinyu970226.Jun 11 2018, 3:47 AM

Change 429211 abandoned by Arturo Borrero Gonzalez:
labs_bootstrapvz: firstboot.sh: bring back some resolv.conf magic

Reason:
It seems this is no longer relevant.