bd808@deployment-shellbox01:~$ sudo -i puppet agent -tv Info: Using environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Function lookup() did not find a value for the name 'profile::tlsproxy::envoy::upstream_tls' (file: /srv/puppet_code/environments/production/modules/profile/manifests/tlsproxy/envoy.pp, line: 75) on node deployment-shellbox01.deployment-prep.eqiad1.wikimedia.cloud Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run
Failure appears to be caused by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1219770 where @ABran-WMF introduced new Hera settings profile::tlsproxy::envoy::upstream_tls and profile::tlsproxy::envoy::upstream_sni. These have been given Production global defaults in hieradata/common/profile/tlsproxy/envoy.yaml, but that config file is not in the puppetserver's hiera lookup scope within the Beta Cluster or other WMCS projects.
A local fix for Beta Cluster is copying the new hiera settings into the project global hiera:
https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/e1aea5f956fbc945bd7bb34ede02b24e77efbaf5%5E%21/#F0
diff --git a/deployment-prep/_.yaml b/deployment-prep/_.yaml index 2151103..5f2d748 100644 --- a/deployment-prep/_.yaml +++ b/deployment-prep/_.yaml @@ -840,6 +840,8 @@ eqiad: mw_media: abracadabra profile::tlsproxy::envoy::ssl_provider: cfssl +profile::tlsproxy::envoy::upstream_sni: null +profile::tlsproxy::envoy::upstream_tls: false prometheus_nodes: - deployment-prometheus05.deployment-prep.eqiad1.wikimedia.cloud puppetdb::password::replication: ''
bd808@deployment-shellbox01:~$ sudo -i puppet agent -tv Info: Using environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Loading facts Info: Caching catalog for deployment-shellbox01.deployment-prep.eqiad1.wikimedia.cloud Info: Applying configuration version '(c915f3960a) gitpuppet - varnish: Move error message from footer to body for HTTP 4xx responses' Notice: /Stage[main]/Apt/File[/etc/apt/sources.list]/ensure: removed (corrective) Notice: /Stage[main]/Profile::Docker::Runner/Service::Docker[shellbox]/Systemd::Service[shellbox]/Service[shellbox]/ensure: ensure changed 'stopped' to 'running' (corrective) Info: /Stage[main]/Profile::Docker::Runner/Service::Docker[shellbox]/Systemd::Service[shellbox]/Service[shellbox]: Unscheduling refresh on Service[shellbox] Notice: Applied catalog in 17.73 seconds