maurelio@deployment-mira:/etc/puppet$ cd modules
-bash: cd: modules: No such file or directory

It makes sense therefore that puppet can't find the files:

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Could not find data item profile::kubernetes::deployment_server::git_owner in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/kubernetes/deployment_server.pp:5:16 on node deployment-mira.deployment-prep.eqiad.wmflabs

@Joe @Dzahn @bd808 Any idea? Thanks.

@MarcoAurelio This looks like it's about data missing in Hiera. In production we have:

hieradata/role/common/deployment_server.yaml:profile::kubernetes::deployment_server::git_owner: trebuchet

It would probably go away if that was set in Hiera for deployment_prep project.

Change 423178 had a related patch set uploaded (by MarcoAurelio; owner: MarcoAurelio):
[labs/private@master] hieradata: fix for deployment-tin/mira lack of ::git_owner

https://gerrit.wikimedia.org/r/423178

Change 423178 merged by Dzahn:
[labs/private@master] hieradata: fix for deployment-tin/mira lack of ::git_owner

https://gerrit.wikimedia.org/r/423178

Puppet still failing:

maurelio@deployment-tin:~$ sudo puppet agent -tv
Info: Using configured environment 'future'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Could not find data item profile::kubernetes::deployment_server::git_owner in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/kubernetes/deployment_server.pp:5:16 on node deployment-tin.deployment-prep.eqiad.wmflabs
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Not sure what's going on. Beta is Release-Engineering-Team turf so maybe someone could take a look here and fix not only this one but the whole lot at http://shinken.wmflabs.org/problems ? I've been asking around for weeks, yet no one from RelEng seems to care about those errors? That's not okay.

Change 423232 had a related patch set uploaded (by EddieGP; owner: EddieGP):
[operations/puppet@production] hiera: fix deployment-mira, lacking ::git_owner

https://gerrit.wikimedia.org/r/423232

Change 423232 merged by Dzahn:
[operations/puppet@production] hiera: fix deployment-mira, lacking ::git_owner

https://gerrit.wikimedia.org/r/423232

Still erroring:

maurelio@deployment-mira:~$ sudo puppet agent -tv
Info: Using configured environment 'future'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Could not find data item profile::kubernetes::deployment_server::git_group in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/kubernetes/deployment_server.pp:6:16 on node deployment-mira.deployment-prep.eqiad.wmflabs
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Change 423256 had a related patch set (by Paladox) published:
[operations/puppet@production] hiera: fix deployment-mira, lacking ::git_group

https://gerrit.wikimedia.org/r/423256

Change 423256 merged by Dzahn:
[operations/puppet@production] hiera: fix deployment-mira, lacking ::git_group

https://gerrit.wikimedia.org/r/423256

this also fixed puppet runs on a bunch of other deployment-* hosts thanks to using common.yaml instead of ./hosts/ bonus token for that ! thanks

15:48 < shinken-wm> RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0]
16:36 < shinken-wm> RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0]
17:05 < shinken-wm> RECOVERY - Puppet staleness on deployment-eventlog05 is OK: OK: Less than 1.00% above the threshold [3600.0]
17:52 < shinken-wm> RECOVERY - Puppet staleness on deployment-maps03 is OK: OK: Less than 1.00% above the threshold [3600.0]