Today I was running clush -w @all 'sudo puppet agent --test' from the host tools-clushmaster-01.eqiad.wmflabs and some issues happened in the output.
In case of tools-bastion-05.tools.eqiad.wmflabs, this was:
[...]
tools-webgrid-lighttpd-1409.tools.eqiad.wmflabs: Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
tools-webgrid-generic-1401.tools.eqiad.wmflabs: Notice: Finished catalog run in 73.50 seconds
tools-exec-1417.tools.eqiad.wmflabs: Info: Retrieving pluginfacts
tools-bastion-05.tools.eqiad.wmflabs: Notice: /Stage[main]/Diamond/Service[diamond]: Triggered 'refresh' from 1 events
tools-exec-1417.tools.eqiad.wmflabs: Info: Retrieving plugin
tools-exec-1409.tools.eqiad.wmflabs: Notice: Finished catalog run in 95.50 seconds
tools-bastion-05.tools.eqiad.wmflabs: Notice: /Stage[main]/Prometheus::Node_puppet_agent/Cron[prometheus_puppet_agent_stats]/ensure: created
tools-bastion-05.tools.eqiad.wmflabs: Permission denied (publickey,hostbased).
tools-bastion-05.tools.eqiad.wmflabs:
tools-bastion-05.tools.eqiad.wmflabs: NOTE: some crontab entries have been modified to grid submissions.
tools-bastion-05.tools.eqiad.wmflabs: You may want to examine the result with 'crontab -e'.
tools-bastion-05.tools.eqiad.wmflabs:
tools-bastion-05.tools.eqiad.wmflabs: Notice: Finished catalog run in 116.87 seconds
tools-exec-1417.tools.eqiad.wmflabs: Info: Loading facts in /var/lib/puppet/lib/facter/interface_primary.rb
tools-exec-1417.tools.eqiad.wmflabs: Info: Loading facts in /var/lib/puppet/lib/facter/puppet_settings.rb
[...]
There seems to be some issue with ssh keys or host authorization.