A straw-man proposal:
Each VM definitely knows about one puppet 'core' server which only serves up the base config.
- no custom classes
- never a project-local master, only ever a central/standard master that manages all VMs
- reads user-defined custom hiera, but sanitizes it for a whitelist of settings which can't break the catalog
- For starters, only settings allowed are: custom_puppet_master, lvs_mount things, cert-switching
Optionally, a given VM can also subscribe to a second 'custom' master, which might be project-local or might be an additional 'central' master.
- this is almost entirely orthogonal to the 'core' puppet run. Different config dir, different certs, run triggered by a different cron, etc.
- default VMs don't know about this server at all
- /only/ serves classes specified by the custom enc, empty by default
- able to read the entire custom hiera catalog (or possibly an separate unrelated hiera catalog)
Q: what if a config state in the custom config contradicts a config state in the core config?
A: VM state will oscillate between the two depending on which puppet run happened most recently. This won't happen very often (minimal overlap in the scope of the two puppet masters) but when it does it will be the user's problem (NOTE: this is moot if the custom puppet catalog is always a superset of the core catalog)
Q: what will the UI look like for this?
A: To be determined but not necessarily a lot different from the current UI. We'll still want a way to add project-wide custom puppet for some use cases. (Probably the whole puppet UI needs redesigning but most of that is decoupled from this issue)
Q: What about module versioning?
A: I'm pretty sure that versioning is a good idea, and also pretty sure that it's unrelated to the issue that this proposal is meant to fix, which is that even if the base module is rock solid you can still break the catalog by adding literally anything else to the catalog for a given puppet run.