Page MenuHomePhabricator

Add a ssh key pair for scap on beta cluster
Closed, ResolvedPublic

Description

Release engineering is migrating scap deployment from a Debian package to scap T303559

On the Beta-Cluster-Infrastructure we would need a keyholder entry in order to deploy scap with scap. Production has:

ssh::userkey { 'scap':
  content => secret('keyholder/scap.pub'),
}

Event Timeline

Change 804568 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] deployment-prep: add keyholder agent for scap

https://gerrit.wikimedia.org/r/804568

Thanks for this task @hashar

The Puppet config

ssh::userkey { 'scap':
  content => secret('keyholder/scap.pub'),
}

also applies to the beta cluster and creates an entry here:

jnuche@deployment-deploy03:~$ cat /etc/ssh/userkeys/scap
ssh-rsa SNAKEOIL scap

I think that's the public key used by ssh to grant access from the remote host but it's not the id used by keyholder. The keyholder id lives in production at: /etc/keyholder.d/scap.pub and that's the one missing on the beta cluster deploy host:

jnuche@deployment-deploy03:~$ ls /etc/keyholder.d/scap.pub
ls: cannot access '/etc/keyholder.d/scap.pub': No such file or directory

From the quick tour I gave to Jaime, the beta cluster deployment server is the WMCS instance deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud which is configured to use the Puppet master deployment-puppetmaster04.deployment-prep.eqiad.wmflabs on which we have full root.

That puppet master is described at https://wikitech.wikimedia.org/wiki/Help:Standalone_puppetmaster and we can apply changes to /var/lib/operations/puppet to test puppet manifest. Secrets are held in patches local to the instance under /var/lib/labs/private and are not pushed back to Gerrit.

https://wikitech.wikimedia.org/wiki/Keyholder describes the Keyholder and lists keys available on the Beta Cluster. They are stored on deployment-puppetmaster04.deployment-prep.eqiad.wmflabs in /var/lib/git/labs/private. It is a clone of the PUBLIC repository https://gerrit.wikimedia.org/g/labs/private for which https://gerrit.wikimedia.org/r/c/788677 introduced a dummy ssh key pair (T307351).

The next step is to generate a ssh keypair on deployment-puppetmaster04 to replace the dummy pair and add it as a local commit:

deployment-puppetmaster04:/var/lib/git/labs/private(master u+40)# ssh-keygen -t ed25519 -b 256 -f modules/secret/secrets/keyholder/scap
Generating public/private ed25519 key pair.
modules/secret/secrets/keyholder/scap already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in modules/secret/secrets/keyholder/scap.
Your public key has been saved in modules/secret/secrets/keyholder/scap.pub.
...
$ git add modules/secret/secrets/keyholder/scap modules/secret/secrets/keyholder/scap.pub
$ git commit ...

I have stored the passphrase in deployment-puppetmaster04:/var/lib/git/labs/private/files/ssh/tin/scap.passphrase and added that location to the doc https://wikitech.wikimedia.org/w/index.php?title=Keyholder&diff=1988289&oldid=1934855

I ran puppet on deployment-deploy03 which has installed the ssh keypair.

I crafted https://gerrit.wikimedia.org/r/c/operations/puppet/+/804568 to configure the Keyholder agent to recognize scap and cherry picked it on deployment-puppetmaster04 under /var/lib/git/operations/puppet:

git fetch https://gerrit.wikimedia.org/r/operations/puppet refs/changes/68/804568/1 && git cherry-pick FETCH_HEAD

Ran puppet on deployment-deploy03 which created:

/etc/keyholder.d/scapthe private key
/etc/keyholder.d/scap.pubthe public key
/etc/keyholder-auth.d/scap.ymltrusted policy

I then added the key to keyholder:

$ sudo keyholder add /etc/keyholder.d/scap
Enter passphrase for /etc/keyholder.d/scap: 
Identity added: /etc/keyholder.d/scap (root@deployment-puppetmaster04)
$ keyholder status
keyholder-agent: active
...
- 256 SHA256:mbI0ELyh+jn9O2h0RPd3b56T5skD92kyYz/osf4sn5c root@deployment-puppetmaster04 (ED25519)
keyholder-proxy: active
...
- 256 SHA256:mbI0ELyh+jn9O2h0RPd3b56T5skD92kyYz/osf4sn5c root@deployment-puppetmaster04 (ED25519)

I am not sure why it shows the key comment rather than the path to the private key though.

$ keyholder status
keyholder-agent: active
- 256 SHA256:mbI0ELyh+jn9O2h0RPd3b56T5skD92kyYz/osf4sn5c root@deployment-puppetmaster04 (ED25519)
keyholder-proxy: active
- 256 SHA256:mbI0ELyh+jn9O2h0RPd3b56T5skD92kyYz/osf4sn5c root@deployment-puppetmaster04 (ED25519)

I am not sure why it shows the key comment rather than the path to the private key though.

See T257329, ssh shows the key comment rather than the file, thus the key has to be generated with a comment showing the path:

ssh-keygen -C "/etc/keyholder.d/scap"

I have changed it with ssh-keygen -c -C '/etc/keyholder.d/scap' -f /vaamer/lib/git/labs/private/modules/secret/secrets/keyholder/scap

Ran puppet on deployment-deploy03 then keyholder add /etc/keyholder.d/scap and this time it looks fine:

$ /usr/local/sbin/keyholder status
keyholder-agent: active
...
- 256 SHA256:mbI0ELyh+jn9O2h0RPd3b56T5skD92kyYz/osf4sn5c /etc/keyholder.d/scap (ED25519)
keyholder-proxy: active
...
- 256 SHA256:mbI0ELyh+jn9O2h0RPd3b56T5skD92kyYz/osf4sn5c /etc/keyholder.d/scap (ED25519)

The key has been added, task is pending for the Puppet change to configure the keyholder which is https://gerrit.wikimedia.org/r/c/804568

Change 804568 merged by Jbond:

[operations/puppet@production] deployment-prep: add keyholder agent for scap

https://gerrit.wikimedia.org/r/804568