Page MenuHomePhabricator

Scap: keyholder Too many authentication failures
Closed, ResolvedPublic

Description

scap uses ssh keys inside keyholder serially to request access to targets. The number of keys in an agent is limited by MaxAuthTries which is a security feature in sshd_config(5) that limits the number of authentication attempts per connection.

We've upped the limit of MaxAuthTries once, but a fix inside scap would be ideal.

Event Timeline

thcipriani triaged this task as Medium priority.
thcipriani moved this task from Needs triage to Debt on the Scap board.

Change 369817 had a related patch set uploaded (by Thcipriani; owner: Thcipriani):
[operations/puppet@production] keyholder: public keys publicly readable

https://gerrit.wikimedia.org/r/369817

Change 369817 merged by Filippo Giunchedi:
[operations/puppet@production] keyholder: public keys publicly readable

https://gerrit.wikimedia.org/r/369817

Now that scap 3.7.0-1 is live, this should be fixed. To use a specific keyholder key, use the keyholder_key configuration value with the value that is the same as key_name for scap::target in puppet. e.g., for gerrit:

jetty.pp
scap::target { 'gerrit/gerrit':
  deploy_user: 'gerrit2',
  key_name: 'gerrit',
}
gerrit/scap/scap.cfg
keyholder_key: gerrit

To test that a key is working with keyholder before deployment use the ssh command:

SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -v -o BatchMode=yes -o User=[username] -o IdentitiesOnly=yes -o IdentityFile=/etc/keyholder.d/[keyname].pub [hostname]

Change 376734 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/software/librenms@master] Force the ssh key to be used by scap

https://gerrit.wikimedia.org/r/376734

Change 376734 merged by Alexandros Kosiaris:
[operations/software/librenms@master] Force the ssh key to be used by scap

https://gerrit.wikimedia.org/r/376734

Change 376735 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Revert "sshd_config: Increase MaxAuthTries"

https://gerrit.wikimedia.org/r/376735

Change 376735 merged by Alexandros Kosiaris:
[operations/puppet@production] Revert "sshd_config: Increase MaxAuthTries"

https://gerrit.wikimedia.org/r/376735

Change 377268 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Revert "Revert "sshd_config: Increase MaxAuthTries""

https://gerrit.wikimedia.org/r/377268

Change 377268 merged by Alexandros Kosiaris:
[operations/puppet@production] Revert "Revert "sshd_config: Increase MaxAuthTries""

https://gerrit.wikimedia.org/r/377268

Change 377269 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Revert "Revert "Revert "sshd_config: Increase MaxAuthTries"""

https://gerrit.wikimedia.org/r/377269

Change 377966 had a related patch set (by Alexandros Kosiaris) published:
[mediawiki/services/parsoid/deploy@master] Remove the dsh-targets file and use the dsh group. That file is automatically generated from confd and is guaranteed to be up to date, which is not the case for the dsh-targets file.

https://gerrit.wikimedia.org/r/377966

Change 377966 merged by Alexandros Kosiaris:
[mediawiki/services/parsoid/deploy@master] Remove the dsh-targets file and use the dsh group.

https://gerrit.wikimedia.org/r/377966

Change 380503 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Specify keyholder_key in global scap.cfg

https://gerrit.wikimedia.org/r/380503

Change 377269 merged by Alexandros Kosiaris:
[operations/puppet@production] Revert "Revert "Revert "sshd_config: Increase MaxAuthTries"""

https://gerrit.wikimedia.org/r/377269

https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/377269/ had fallen through the cracks. It's now merged, right before a SWAT window, in order to identify issues as fast as possible

Change 380503 abandoned by Alexandros Kosiaris:
Specify keyholder_key in global scap.cfg

https://gerrit.wikimedia.org/r/380503

This is happening to me on phab1001 now and phab's scap::target includes the requisite part:

scap::target { $deploy_target:
    deploy_user => $deploy_user,
    key_name    => 'phabricator',

Am I missing something? Does this need to be reopened? For some reason I never hit that error ("Too many authentication failures") with phabricator scap deployments until now.

This is happening to me on phab1001 now and phab's scap::target includes the requisite part:

scap::target { $deploy_target:
    deploy_user => $deploy_user,
    key_name    => 'phabricator',

Am I missing something? Does this need to be reopened? For some reason I never hit that error ("Too many authentication failures") with phabricator scap deployments until now.

Do you have keyholder_key in your scap.cfg? IIRC, it tries to use a key named after your ssh-user if keyholder_key isn't set.

So I'm dumb. I had keyholder_key set in puppet but not in the scap.cfg in the deployment repo. False alarm, sorry! (Tested it, works now)