Page MenuHomePhabricator

Define which SSH key to use to SSH from the new cloud-cumin to hosts
Closed, ResolvedPublic

Description

Can we re-use the existing one from the Cloud Cumin hosts (e.g. cloud-cumin-03.cloudinfra.eqiad1.wikimedia.cloud)?

Event Timeline

fnegri renamed this task from Define which user and SSH key to use to SSH to the new Cumin VMs to Define which SSH key to use to SSH from the new cloud-cumin to hosts.Nov 23 2022, 2:09 PM
fnegri updated the task description. (Show Details)

Note that the raw private key has been exposed to people without production root (like me), so while the key is probably good for SSHing to cloud hosts, you probably don't want to re-use it for production access.

@taavi indeed, they will be two separate keys.

I've just discussed this with @dcaro, and this is my current understanding:

  • the new Ganeti-based cloud-cumins (maybe we should find a better name, to avoid confusion with the CloudVPS cloud-cumins) will need to SSH both to CloudVPS VMs and to physical WMCS hosts (e.g. cloudvirt, cloudceph, etc.)
  • we can add two private SSH keys to the Keyholder on the Ganeti cloud-cumins, one key will have access to CloudVPS VMs and the second key will have access to the physical WMCS hosts
    • the first SSH key could be the existing cumin_openstack_master key
    • the second one will be a new key, that will be created following this procedure, and its password will be stored in Pwstore
  • Keyholder can be configured to allow a specific LDAP group (which one?) to use the key while logged into the new Ganeti cloud-cumins

@taavi @Volans does this look good to you?

Keyholder can be configured to allow a specific LDAP group (which one?) to use the key while logged into the new Ganeti cloud-cumins

In production it can be controlld based unix groups controlled by the admin Puppet module, not LDAP groups directly. I imagine we probably want wmcs-roots and ops.

LGTM. @fnegri please lmk if you want to change the hostnames and which to use before we create the VMs to reduce unnecessary work ;)

@taavi thanks, I'm always confused by the different groups, and was looking for wmcs-roots in LDAP without finding it 😬

@Volans (and others) let's brainstorm possible hostnames for the Ganeti VMs:

  • cloud-cumin
  • cloud-cookbooks
  • wmcs-cumin (this is my favourite, but I might change my mind tomorrow :P)
  • wmcs-cookbooks

If you can think of other names that would make sense to you, write them below!

Since all other production hosts use cloud as a prefifx I'd prefer to avoid using wmcs for a single host or two.

fnegri changed the task status from Open to In Progress.Nov 24 2022, 3:21 PM
fnegri claimed this task.
fnegri triaged this task as High priority.

Makes sense to stick with cloud, what about using cloud-cookbooks then, to avoid having the same cloud-cumin prefix for both cloud-cumin-xx.cloudinfra.eqiad1.wikimedia.cloud (Cloud VPS VMs) and cloud-cuminYYYY.eqiad.wmnet (Ganeti VMs)?

I can live with using cloud-cumin for both, but I'm worried it could be confusing. On the other hand, it's possible that once we have the new Ganeti VMs, we might not need the Cloud VPS cumin VMs anymore.

After some research, there are 2 things I'm confused about:

  1. we have two versions of the cumin_openstack_master key, one is maintained in labs/private (pub key ending with K27jK), the other one is in Private Puppet. The first one is the one that ends up in /etc/ssh/userkeys/root.d/cumin in CloudVps VMs. Am I correct that the second key is not used anywhere? Can it be removed?
  1. The new key that we want to generate for this task (I will call it cloud_cumin_master unless someone can think of a better name) will have to be added to /etc/ssh/userkeys/root.d only for cloud-related production hosts (e.g. cloudcephmon1001.eqiad.wmnet) but not for other productions hosts. I don't think we have any way to select those in Puppet, should we create a new Puppet class, something like profile::cloud that will be included in all the hosts we want to reach from the new Cloud Cumins?

This all sounds good to me
Just echoing some comments from IRC

  1. ... but not for other productions hosts. I don't think we have any way to select those in Puppet, should we create a new Puppet class, something like profile::cloud that will be included in all the hosts we want to reach from the new Cloud Cumins?

We dont have an easy way to do this, i proposed to riccardo today that we abuse the regex.yaml part of yaml This has the benefit that users shouldn't need to make changes here when future roles are introduced (assuming the name starts with cloud). It would also be possible to go down the rout of creating some cloud specific profile. if you where to do this i would recommend creating profile::base::cloud and have that include profile::base::production. you could then update all the cloud roles with sed 's/profile::base::production/profile::base::cloud/ which should at least mean that future roles have a better chance of including the correct base profile as they will just copy paste from some current role

Change 861855 had a related patch set uploaded (by FNegri; author: jbond):

[operations/puppet@production] WIP: idea for cloud cumin::target

https://gerrit.wikimedia.org/r/861855

About the hostname, given there was no clear consensus on other options, I suggest we proceed with cloud-cuminXXXX.eqiad.wmnet, unless there are any objections.

Please note this hostname would also fall under the ^cloud regex, and the new VMs will become a cumin target for themselves (i.e. they will be accessible with the new cloud_cumin_master SSH key), but I don't think that will cause any issue.

About the hostname, given there was no clear consensus on other options, I suggest we proceed with cloud-cuminXXXX.eqiad.wmnet, unless there are any objections.

I prefer cloudcuminXXXX.eqiad.wmnet as we don't hyphenate any of our other hostnames.

Makes sense, let's use cloudcuminXXXX.eqiad.wmnet then!

I created the new SSH key following this procedure, the key name is cloud_cumin_master. I added the password for the key to pwstore as cloud-cumin-master-key-passphrase.

Change 865047 had a related patch set uploaded (by FNegri; author: FNegri):

[labs/private@master] Rename cloudcumin key to match production name

https://gerrit.wikimedia.org/r/865047

Change 865047 merged by FNegri:

[labs/private@master] Rename cloudcumin key to match production name

https://gerrit.wikimedia.org/r/865047

Change 867180 had a related patch set uploaded (by Volans; author: Volans):

[labs/private@master] keyholder: adjust comment for cloud_cumin_master

https://gerrit.wikimedia.org/r/867180

Change 867180 merged by FNegri:

[labs/private@master] keyholder: adjust comment for cloud_cumin_master

https://gerrit.wikimedia.org/r/867180

Change 867595 had a related patch set uploaded (by FNegri; author: FNegri):

[labs/private@master] Add "snakeoil" private key cloud_cumin_master

https://gerrit.wikimedia.org/r/867595

Change 867595 merged by FNegri:

[labs/private@master] Add "snakeoil" private key cloud_cumin_master

https://gerrit.wikimedia.org/r/867595

Change 861855 abandoned by FNegri:

[operations/puppet@production] cumin::target: Add support for cloudcumin hosts

Reason:

Abandoning in favor of https://gerrit.wikimedia.org/r/c/operations/puppet/+/867169

https://gerrit.wikimedia.org/r/861855

Change 868655 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] cumin::cloud_target: read also the cloud_cumin key

https://gerrit.wikimedia.org/r/868655

Change 868655 merged by Volans:

[operations/puppet@production] cumin::cloud_target: read also the cloud_cumin key

https://gerrit.wikimedia.org/r/868655

Change 868732 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] Use a single file for public key

https://gerrit.wikimedia.org/r/868732

Change 874859 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] cumin::target: use concat to manage the file

https://gerrit.wikimedia.org/r/874859

Change 868732 merged by FNegri:

[operations/puppet@production] Make sure cloud_cumin public key is evaluated

https://gerrit.wikimedia.org/r/868732

Change 874859 abandoned by Jbond:

[operations/puppet@production] cumin::target: use concat to manage the file

Reason:

not required

https://gerrit.wikimedia.org/r/874859