Developing custom Puppet manifests is a major pain point for Cloud VPS project owners who are not working directly with a member of the Wikimedia TechOps team. This is especially painful for projects that are not developing/testing deployments which are eventually destined for Wikimedia's production cluster.
The current best practice for these types of projects is to run a project local Puppetmaster and keep a collection of 1-N local patches that are rebased on top of the production ops/puppet.git branch. This has several drawbacks:
- Rebases before merge of upstream of cherry-picked patches can break the automatic rebase cron
- Local only patches are subject to loss in the event that the Puppetmaster instance is destroyed
- Project local changes are not easily visible for copying to other projects or review outside of the project maintainers
- Project local changes are not grep'able for people making upstream changes
- Switching from one puppetmaster to another requires manual intervention (T152941)
Puppet has a feature called environments that is in part intned to solve the use case where a single Puppetmaster is providing manifests to multiple clients with differing maintainers. The feature allows for a shared collection of manifests that are used in all environments and isolated manifest/module trees selected by the client's configuration.
This setup may not be a useful solution for the use-case of pre-merge testing of cherry-picks from ops/puppet.git (e.g. deployment-prep), but it sounds promising for providing a means for a Cloud VPS project to manage a collection of Puppet manifests/modules that sit side-by-side with the core ops/puppet.git collection. Implementation sounds like it would roughly be cloning the additional Puppet manifests on the shared Pupeptmaster and then setting the project or instance to use the new environment. The per-environment Puppet resources could be manged in either Diffusion or Gerrit.
This hypothesis needs proof of concept testing and security review to ensure that environment X can not leak into environment Y nor steal secrets from other environments.