Page MenuHomePhabricator

Work out what we're going to do with the cumin master functionality currently on labpuppetmaster1001
Closed, ResolvedPublic

Description

In the parent ticket we're going to be creating a central puppetmaster instance(s) in the cloudinfra project.
One of the functions of the current central puppetmaster setup is Cumin. Cumin's private key lives here, presumably via production's puppet-private secrets, and it SSHes into target VMs using bastion-restricted as a proxy.
There's a few unanswered questions around this:

  • Why is labpuppetmaster1001 involved at all? Why not just have bastion-restricted be a Cumin master? Are there concerns around key security if cumin lives on an instance too exposed to the internet like bastion-restricted? I know in the past there have been issues causing bastion-restricted to ironically be one of the least restricted Wikimedia servers.
  • Why are various unrelated cloudinfra instances set up like bastion-restricted to get connections directly from labpuppetmaster1001 instead of just bastion-restricted? Specifically it's the profile::openstack::main::cumin::auth_group: cumin_real_masters in their hiera. Additionally it's only a subset of cloudinfra - the puppetmasters Andrew made in there appear to be normal.
  • Should the Cumin master for Cloud VPS continue to be hosted alongside the central puppetmaster, or should it be a separate instance?

Event Timeline

Hopefully @Andrew can shed some light here

I don't know a ton about how/why things are set up the way they are currently (even though much of it is doubtless my fault.) The idea of having private/secure/important/cloud-wide things on a VM is still a pretty new idea, so the answer to most "why is it this way" questions will probably be related to that.

I'm pretty sure that what I want is a single stand-alone VM in cloudinfra that has the ability to take cumin actions on every VM. It seems like this should be possible/simple but I may be overlooking things. I don't think there's any reason to conflate this issue with the puppetmaster rebuild; they're currently colocated in prod to safe on hardware.

I think the expectation will be once puppetmaster gets moved to get rid of the prod hardware... Like "we have this labpuppetmaster1001, but didn't that get moved into labs? oh, it doesn't run puppetmaster anymore, it's just the cumin master.... ???". Reminds me of how for a while deployment-salt was the deployment-prep puppetmaster. So I figured I'd make a subtask about this and consider it part of the work.
Could move this to being a direct child of T207536: Move various support services for Cloud VPS currently in prod into their own instances?

+1 to having a cumin specific server in the cloudinfra project. Hopefully, enabled with spicerack/cookbook as I commented on IRC.

This means 2 different servers, right?

  • wmcs-puppetmater-01.cloudinfra.eqiad.wmflabs (or whatever is the name), i.e, the central puppetmaster
  • wmcs-cumin-01.cloudinfra.eqiad.wmflabs (or whatever is the name), i.e, the central cumin server for all VMs in Cloud VPS

It could do, I feel it should be separate, Andrew feels it should be separate. Currently it's not separate though, and IIRC there's one random NFS-related script that expects them to be on the same box, but I think we could totally fix that.

Change 502202 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] labs central puppetmaster: Allow cumin functionality to be disabled

https://gerrit.wikimedia.org/r/502202

Change 502202 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] labs central puppetmaster: Allow cumin functionality to be disabled

https://gerrit.wikimedia.org/r/502202

Change 502219 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] labs cumin: Allow running nfs_hostlist script outside a puppetmaster

https://gerrit.wikimedia.org/r/502219

Change 502219 merged by Bstorm:
[operations/puppet@production] labs cumin: Allow running nfs_hostlist script outside a puppetmaster

https://gerrit.wikimedia.org/r/502219