root@deployment-salt:/var/lib/git/operations/puppet (production) # puppet agent -tv Warning: Unable to fetch my node definition, but the agent run will continue: Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: i-0000015c.eqiad.wmflabs] Info: Retrieving plugin Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: i-0000015c.eqiad.wmflabs] Error: /File[/var/lib/puppet/lib]: Could not evaluate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: i-0000015c.eqiad.wmflabs] Could not retrieve file metadata for puppet://deployment-salt.eqiad.wmflabs/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: i-0000015c.eqiad.wmflabs] Info: Loading facts in /etc/puppet/modules/base/lib/facter/ec2id.rb Info: Loading facts in /etc/puppet/modules/base/lib/facter/physicalcorecount.rb Info: Loading facts in /etc/puppet/modules/base/lib/facter/initsystem.rb Info: Loading facts in /etc/puppet/modules/base/lib/facter/lldp.rb Info: Loading facts in /etc/puppet/modules/apt/lib/facter/apt.rb Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/pe_version.rb Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/root_home.rb Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/puppet_vardir.rb Info: Loading facts in /etc/puppet/modules/ganeti/lib/facter/ganeti.rb Info: Loading facts in /etc/puppet/modules/puppet_statsd/lib/facter/puppet_config_dir.rb Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb Info: Loading facts in /var/lib/puppet/lib/facter/puppet_config_dir.rb Info: Loading facts in /var/lib/puppet/lib/facter/ec2id.rb Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb Info: Loading facts in /var/lib/puppet/lib/facter/ganeti.rb Info: Loading facts in /var/lib/puppet/lib/facter/apt.rb Info: Loading facts in /var/lib/puppet/lib/facter/physicalcorecount.rb Info: Loading facts in /var/lib/puppet/lib/facter/initsystem.rb Info: Loading facts in /var/lib/puppet/lib/facter/lldp.rb Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: i-0000015c.eqiad.wmflabs] Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run Error: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: i-0000015c.eqiad.wmflabs] root@deployment-salt:/var/lib/git/operations/puppet (production) #
Description
Description
Related Objects
Related Objects
- Mentioned In
- T95273: integration labs project DNS resolver improperly switched to openstack-designate
T95564: /var/lib/l10nupdate fills up deployment-bastion /var partition - Mentioned Here
- P500 changes from puppet run
T95273: integration labs project DNS resolver improperly switched to openstack-designate
Event Timeline
Comment Actions
The puppet failure where due to the hostname of the puppetmaster changing. That causes puppetmaster self to no more recognize the master as being the master and alter the puppet.conf to remove the [master] section. The puppetmaster process is still around though, it ends up with the SSL cert of the client which is in [main] section.
accurately describe the diff that happened.Integration had the exact same issue: T95273.
Comment Actions
Ok solved! That was the exact same issue as on integration and staging project. Changing the hostname cause the puppetmaster to be reverted to a simple client and borks everything.
Fix is to reverse patch puppet.conf and specially the [master] section should have ssldir = /var/lib/puppet/server/ssl (note: /server/)
puppetmaster was deadlocked somehow. Had to kill -9 it.
All went back fine once it restarted with the proper conf.