Page MenuHomePhabricator

Automate beta scap3/keyholder setup
Open, NormalPublic

Description

There are a series of recurring avoidable problems in beta whenever new users move to scap3.

  1. Each user must be added to the deploy-service or other project otherwise you get Agent admitted failure to sign key
  2. Host keys must be accepted for any new hosts that are being SSH'd to by this user/deployment-tin for the first time

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 2 2016, 11:00 PM
bd808 added a comment.Sep 2 2016, 11:03 PM

In my Striker project I solved the group thing by putting this in hiera:

scap::server::keyholder_agents:
    deploy-service:
        trusted_groups:
            - wikidev

For ssh host keys there is T72792: Set up puppet exported resources to collect ssh host keys for beta. Could get the puppet agent to collect them on each host and on the deployment server generate the global ssh know_hosts file. That is afaik how it is done in prod.

So if you use resource collection and care about security (you could block access to puppetmaster from outside the project with security groups), you should turn off auto-signing. Tools runs a limited puppetmaster setup with autosigning turned off, for example.

hashar triaged this task as Normal priority.Sep 12 2016, 8:01 AM
hashar moved this task from To Triage to Backlog on the Beta-Cluster-Infrastructure board.
thcipriani moved this task from Needs triage to Debt on the Scap board.Sep 13 2016, 4:01 PM

what if we created a CA and signed all the host keys with that? We could then have the clients verify the signature based on the CA's signing key instead of having to manually trust each host key.

bd808 added a comment.Sep 13 2016, 8:34 PM

what if we created a CA and signed all the host keys with that? We could then have the clients verify the signature based on the CA's signing key instead of having to manually trust each host key.

Managing a CA and key signing seems a lot of work compared to just getting the host keys added to /etc/ssh/ssh_known_hosts on deployment-tin. Honestly new hosts are spun up so infrequently that could just be managed manually by someone.

Honestly new hosts are spun up so infrequently that could just be managed manually by someone.

Done for right now.

Simplest thing that could work—steps:

  1. Log into wikitech and go to "Manage Instances"
  2. Ensure only deployment-prep is selected in the project filter
  3. Open console and use:
var links = $('.novainstanceid a'),
    len = links.length;

for (var i = 0; i < len; i++) {
    console.log(links.eq(i).text());
}
  1. Copy and paste all the hosts out of the console into a file on deployment-tin, mira.deployment-prep, and mira02.deployment-prep. I used /home/thcipriani/beta-hosts
  2. Add all host keys to the /etc/ssh/ssh_known_hosts file
ssh-keyscan -f /home/thcipriani/beta-hosts >> /etc/ssh/ssh_known_hosts

That is a neat trick! And indeed given a complete list of hostnames it is quite trivial to grab the keys.

I am hereby blaming everyone above to eventually have forced me to read ldapsearch man page (that is a good UX use case)

Here one has:

ldapsearch -S dc -LLL -o ldif-wrap=no -x \
   -b 'ou=hosts,dc=wikimedia,dc=org' \
   '(dc=*deployment-prep*)' dc \
   |grep ^dc|sed -e 's/^dc: //'

Result: a sorted list of instances FQDN, ready to be passed via xargs to Tyler ssh-keyscan magic.

Lets put that in a cron managed by puppet and forget about it ?

thcipriani renamed this task from Fixup beta scap3 keyholder problems to Automate beta scap3/keyholder setup.Sep 26 2016, 12:44 AM
greg removed thcipriani as the assignee of this task.May 16 2017, 3:47 PM