Page MenuHomePhabricator

Puppet errors on various deployment-prep hosts
Closed, ResolvedPublic

Description

Puppet has been failing on the following deployment-prep hosts:

  • deployment-maps05.deployment-prep.eqiad.wmflabs
  • deployment-ms-be03.deployment-prep.eqiad.wmflabs
  • deployment-ms-be04.deployment-prep.eqiad.wmflabs
  • deployment-ms-fe02.deployment-prep.eqiad.wmflabs

If these are one-off test servers, please let us know so we can delete them.

Otherwise, could you take a few minutes to fix that? Thank you!

Event Timeline

GTirloni created this task.

deployment-maps05.deployment-prep.eqiad.wmflabs:

$ puppet agent -tv
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
Info: Loading facts
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: deployment-puppetmaster03.deployment-prep.eqiad.wmflabs]

Fixed by following steps from https://wikitech.wikimedia.org/wiki/Help:Standalone_puppetmaster#Step_2:_Setup_a_puppet_client to generate and sign a new client cert.

Still failing, but for a "better" reason:

$ puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Could not find data item profile::maps::apps::redis_server in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/maps/apps.pp:9:21 on node deployment-maps05.deployment-prep.eqiad.wmflabs

Failures on deployment-ms-be03.deployment-prep.eqiad.wmflabs, deployment-ms-be04.deployment-prep.eqiad.wmflabs, & deployment-ms-fe02.deployment-prep.eqiad.wmflabs:

$ puppet agent -tv
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Could not find data item profile::prometheus::statsd_exporter::mappings in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/prometheus/statsd_exporter.pp:3:29 on node deployment-ms-be03.deployment-prep.eqiad.wmflabs
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Fixed ms-be and ms-fe instances by adding profile::prometheus::statsd_exporter::mappings: [] to project hiera.