Page MenuHomePhabricator

Investigate instances not allowing access from cloud-cumin
Open, Needs TriagePublic

Description

Mostly Permission denied (publickey,keyboard-interactive). but some Permission denied (publickey).:
pontoon-builder-01.monitoring.eqiad.wmflabs
pontoon-conf-01.monitoring.eqiad.wmflabs
pontoon-elastic7-01.monitoring.eqiad.wmflabs
pontoon-grafana-01.monitoring.eqiad.wmflabs
pontoon-log-01.monitoring.eqiad.wmflabs
pontoon-logstash7-02.monitoring.eqiad.wmflabs
pontoon-prometheus-01.monitoring.eqiad.wmflabs
pontoon-puppetdb-01.monitoring.eqiad.wmflabs
snuggle-enwiki-02.snuggle.eqiad.wmflabs
canary1015-01.testlabs.eqiad.wmflabs
cdanis-etcd101.puppet.eqiad.wmflabs
pontoon-puppet-01.monitoring.eqiad.wmflabs

Event Timeline

snuggle-enwiki-02.snuggle.eqiad.wmflabs appears to work now, not sure what changed.
canary1015-01.testlabs.eqiad.wmflabs doesn't seem to accept either my normal key nor my root key. @Andrew do you know anything about that? Looking at the console log in Horizon it seems to be doing stuff.
cdanis-etcd101.puppet.eqiad.wmflabs doesn't seem to accept my root key. @CDanis?
the pontoon-*.monitoring.eqiad.wmflabs ones seem to all have production's cumin masters listed as allowed in /etc/ssh/userkeys/root.d/cumin instead of the Cloud VPS ones. Judging by the diff against origin/production at root@pontoon-puppet-01:/var/lib/git/operations/puppet there's some serious puppet work going on here with it's own ENC, hiera.yaml, etc. This hiera.yaml is missing a hierarchy entry for cloud/%{::wmcs_deployment} and therefore does not receive the correct list of cumin masters - instead defaulting back to production's ones from common.yaml.