Page MenuHomePhabricator

Puppet fails on newly created deployment-wikifeeds01
Closed, ResolvedPublic

Description

I recently created a new beta cluster instance to run the wikifeeds service that's being split off from mobileapps.

Puppet fails as follows:

mholloway-shell@deployment-wikifeeds01:~$ sudo puppet agent -tv
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
Info: Loading facts
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]

Event Timeline

That's what always happens, you still have to go through and manually sort out the situation with certs.

Krenair closed this task as Resolved.EditedJun 11 2019, 9:51 PM
Krenair claimed this task.

Ran cd /var/lib/puppet; mv ssl ssl_old; rm /usr/local/share/ca-certificates/Puppet_Internal_CA.crt; nano /usr/local/share/ca-certificates/Puppet_Internal_CA.crt; update-ca-certificates --fresh; puppet agent -tv, gave it a copy of the CA cert from the puppetmaster, signed the new client cert on the puppetmaster. Puppet cert error is gone, there's a new error which would be a different task.

Edit: One error was just a case of apt-get update, ran puppet a few more times and it's now just looking for a nonexistent docker image.

Thank you, @Krenair. I fixed up the service config, so puppet is running cleanly now as well.

Ok. I'm kind of hoping that with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/506873/ and its parent we can make the puppetmaster switch more or less seamless with just the signing step to perform.