Today I was running `clush -w @all 'sudo puppet agent --test'` from the host `tools-clushmaster-02.eqiad.wmflabs` and some issues happened in the output.
In case of `tools-bastion-02.tools.eqiad.wmflabs`, this was:
> tools-exec-1403.tools.eqiad.wmflabs: Notice: /Stage[main]/Diamond/Service[diamond]: Triggered 'refresh' from 1 events
> tools-exec-1403.tools.eqiad.wmflabs: Notice: Finished catalog run in 91.18 seconds
> tools-bastion-02.tools.eqiad.wmflabs: Notice: /Stage[main]/Prometheus::Node_puppet_agent/Cron[prometheus_puppet_agent_stats]/ensure: created
> tools-bastion-02.tools.eqiad.wmflabs: Permission denied (publickey,hostbased).
> tools-bastion-02.tools.eqiad.wmflabs: NOTE: some crontab entries have been modified to grid submissions.
> tools-bastion-02.tools.eqiad.wmflabs: You may want to examine the result with 'crontab -e'.
> tools-cron-01.tools.eqiad.wmflabs: Notice: Finished catalog run in 90.22 seconds
> tools-bastion-02.tools.eqiad.wmflabs: Notice: Finished catalog run in 79.04 seconds
> clush: tools-exec-1403.tools.eqiad.wmflabs: exited with exit code 2
> tools-worker-1020.tools.eqiad.wmflabs: Info: Caching catalog for tools-worker-1020.tools.eqiad.wmflabs
> tools-worker-1020.tools.eqiad.wmflabs: Info: Applying configuration version '1509456151'
> clush: tools-bastion-02.tools.eqiad.wmflabs: exited with exit code 2
> tools-webgrid-lighttpd-1401.tools.eqiad.wmflabs: Info: Caching catalog for tools-webgrid-lighttpd-1401.tools.eqiad.wmflabs
> tools-paws-worker-1002.tools.eqiad.wmflabs: Info: Retrieving pluginfacts
There seem to be some issue with ssh keys or host authorization.
Related to this issue (same clush run): T179388, T179386