Right now secrets are handled by hand, or via project-local puppetmasters. We probably need some kind of better way to roll out project-wide secrets.

Here's a list of use-cases for secret management:

1. Allowing puppet to deploy a key or password (much as is done in WMF production) without that key or password being visible in the labspuppet git repo
2. Changing a key or password project-wide without having to log into each VM one at a time
3. Allowing custom software to access secrets (e.g. API keys) in code without the key persisting anywhere on disk

Here are a couple of proposed solutions:

1. Support barbican APIs within cloud-vps instances via commandline tooling. Users access these apis using application credentials and manage these credentials (and the resulting secrets) on their own. Some rudimentary puppet integration is added to consume an application credential and load secrets into a custom fact.
2. Extend the Horizon puppet panels to include a 'secret hiera' panel; values from that panel are stored in a private database (or possibly single-key encrypted) and only served to VMs that match the project, prefix, or instance associated with the panel.
3. Hope that OpenStack Heat integrates well enough with Barbican to provide a secrets-enabled provisioning system that gets secrets where they need to be on newly-created VMs. Creation of secrets probably be CLI-only; Any web-UI to manage these secrets would have to be created and maintained by WMCS staff.

Implementing #1 is fairly straightforward (and mostly already done in codfw1dev). #2 is not at all straightforward but should be technically possible. #3 requires considerable further research.