We could implement this as such: Initially we need to create a PGP key for the encrypted passwords. The public key would be distributed to the VMs via puppet. The secret key would be stashed in pwstore. Whether to use a passphrase depends on 4), who would run that script and how often?

The freshly generated password would be encrypted by puppet with the public key. This is reliable and secure as long as we don't leak the secret key. For step 4 the script on the puppetmaster needs to have access to the secret key, which it uses to decrypt the password (as correctly described in b).

@MoritzMuehlenhoff -- that sounds just like what I was envisioning. Would you be willing to spoon-feed me the commands I need (e.g. generating a key, encrypting/decrypting, etc)?

Generating the root password locally and passing the (encrypted) root password over the puppet logs doesn't sound that great to me.

Why wouldn't we generate the cleartext password on the puppetmaster instead, then hash it with mkpasswd and pass it to the agent to set /etc/shadow with it as-is, without the agent ever knowing the cleartext?

It should be trivial to do that kind of thing with a parser function or even generate(). Am I missing one of the requirements perhaps?

@faidon, my design was based on things I already know how to do :) You're suggesting puppet gymnastics that I don't have any experience with, can you be more specific?

(Also, for what it's worth, I don't quite understand how the danger of leaking the root password on a VM to the root account on that VM is a security risk, since... anyone with root can just reset the password on a whim anyway.)

So, a simple way to do this would be:

user { 'root':
    password => generate('/usr/local/bin/password_for_labs', $fqdn)
}

Where /usr/local/bin/password_for_labs could be a script in the puppetmaster with the following contents (untested, proof of concept):

#!/bin/sh
INSTANCE=$1
if [ -f /var/cache/root-passwords/$INSTANCE ]; then
  PASSWORD=$(cat /var/cache/root-passwords/$INSTANCE)
else
  PASSWORD=$(pwgen -sy -N 1)
  umask 027
  echo $PASSWORD > /var/cache/root-passwords/$INSTANCE
fi
mkpasswd -m sha-512 $PASSWORD

A more complicated, potentially nicer way to do this would be to write a better script than a dummy shell script, possibly encrypt the passwords on local storage e.g. with a PGP key, and possibly use a parser function (of which we have plenty in the ops/puppet repo) instead of generate().

BTW, trocla (and puppet-trocla) might also be of interest here. I haven't given it a close look, but it might be worth to give it one for this and other use cases.

Oh, I did not know about generate() -- that's clearly better than what I was imagining.

Encryption-wise... I can't convince myself that it's useful. Root access to the puppetmaster already compromises all instance security, so maybe the extra complication (both in writing and in retrieving passwords) is not worth it.

Change 303617 had a related patch set uploaded (by Andrew Bogott):
Create root passwords for labs instances and store passwords on the puppetmaster

https://gerrit.wikimedia.org/r/303617

Change 303617 merged by Andrew Bogott:
Labs: Generate/store root passwords for instances

https://gerrit.wikimedia.org/r/303617

Change 303826 had a related patch set uploaded (by Andrew Bogott):
Include 'whois' package for mkpasswd

https://gerrit.wikimedia.org/r/303826

Change 303828 had a related patch set uploaded (by Andrew Bogott):
Temporarily remove the code to generate labs root passwords.

https://gerrit.wikimedia.org/r/303828

Change 303828 merged by Andrew Bogott:
Temporarily remove the code to generate labs root passwords.

https://gerrit.wikimedia.org/r/303828

Change 303829 had a related patch set uploaded (by Andrew Bogott):
Don't install root password if has_admin = false

https://gerrit.wikimedia.org/r/303829

Change 303829 merged by Andrew Bogott:
Don't install root password if has_admin = false

https://gerrit.wikimedia.org/r/303829

Change 303826 merged by Andrew Bogott:
Fixups for labs password generation:

https://gerrit.wikimedia.org/r/303826