Today I was running clush -w @all 'sudo puppet agent --test' from the host tools-clushmaster-02.eqiad.wmflabs and some issues happened in the output.
In case of tools-bastion-02.tools.eqiad.wmflabs, this was:
tools-exec-1403.tools.eqiad.wmflabs: Notice: /Stage[main]/Diamond/Service[diamond]: Triggered 'refresh' from 1 events
tools-exec-1403.tools.eqiad.wmflabs: Notice: Finished catalog run in 91.18 seconds
tools-bastion-02.tools.eqiad.wmflabs: Notice: /Stage[main]/Prometheus::Node_puppet_agent/Cron[prometheus_puppet_agent_stats]/ensure: created
tools-bastion-02.tools.eqiad.wmflabs: Permission denied (publickey,hostbased).
tools-bastion-02.tools.eqiad.wmflabs: NOTE: some crontab entries have been modified to grid submissions.
tools-bastion-02.tools.eqiad.wmflabs: You may want to examine the result with 'crontab -e'.
tools-cron-01.tools.eqiad.wmflabs: Notice: Finished catalog run in 90.22 seconds
tools-bastion-02.tools.eqiad.wmflabs: Notice: Finished catalog run in 79.04 seconds
clush: tools-exec-1403.tools.eqiad.wmflabs: exited with exit code 2
tools-worker-1020.tools.eqiad.wmflabs: Info: Caching catalog for tools-worker-1020.tools.eqiad.wmflabs
tools-worker-1020.tools.eqiad.wmflabs: Info: Applying configuration version '1509456151'
clush: tools-bastion-02.tools.eqiad.wmflabs: exited with exit code 2
tools-webgrid-lighttpd-1401.tools.eqiad.wmflabs: Info: Caching catalog for tools-webgrid-lighttpd-1401.tools.eqiad.wmflabs
tools-paws-worker-1002.tools.eqiad.wmflabs: Info: Retrieving pluginfacts
There seem to be some issue with ssh keys or host authorization.