Page MenuHomePhabricator

Investigate use of Puppet "environments" for per-project Puppet manifests
Open, MediumPublic

Description

Developing custom Puppet manifests is a major pain point for Cloud VPS project owners who are not working directly with a member of the Wikimedia TechOps team. This is especially painful for projects that are not developing/testing deployments which are eventually destined for Wikimedia's production cluster.

The current best practice for these types of projects is to run a project local Puppetmaster and keep a collection of 1-N local patches that are rebased on top of the production ops/puppet.git branch. This has several drawbacks:

  • Rebases before merge of upstream of cherry-picked patches can break the automatic rebase cron
  • Local only patches are subject to loss in the event that the Puppetmaster instance is destroyed
  • Project local changes are not easily visible for copying to other projects or review outside of the project maintainers
  • Project local changes are not grep'able for people making upstream changes
  • Switching from one puppetmaster to another requires manual intervention (T152941)

Puppet has a feature called environments that is in part intned to solve the use case where a single Puppetmaster is providing manifests to multiple clients with differing maintainers. The feature allows for a shared collection of manifests that are used in all environments and isolated manifest/module trees selected by the client's configuration.

This setup may not be a useful solution for the use-case of pre-merge testing of cherry-picks from ops/puppet.git (e.g. deployment-prep), but it sounds promising for providing a means for a Cloud VPS project to manage a collection of Puppet manifests/modules that sit side-by-side with the core ops/puppet.git collection. Implementation sounds like it would roughly be cloning the additional Puppet manifests on the shared Pupeptmaster and then setting the project or instance to use the new environment. The per-environment Puppet resources could be manged in either Diffusion or Gerrit.

This hypothesis needs proof of concept testing and security review to ensure that environment X can not leak into environment Y nor steal secrets from other environments.

Event Timeline

bd808 renamed this task from Invesitgate use of Puppet "modules" for per-project Puppet manifests to Invesitgate use of Puppet "environments" for per-project Puppet manifests.Jul 13 2017, 7:58 PM
Andrew renamed this task from Invesitgate use of Puppet "environments" for per-project Puppet manifests to Investigate use of Puppet "environments" for per-project Puppet manifests.Nov 27 2019, 4:45 PM
Bstorm triaged this task as Medium priority.Feb 11 2020, 5:26 PM

I had/have a use case where i wanted to use some third party modules as such i have added r10k support and with it dynamic environment support to the puppetmaster::standalone role. Once enabled by adding role::puppetmaster::standalone::use_r10k: true it will by default use the local checked out git repo /var/lib/git/operations/puppet/ for the master branch and then sync any other branches which exist in https://gerrit.wikimedia.org/r/operations/puppet, prefixing each branch with dev.

Currently the operations production rtepo has the following branches

refs/heads/pontoon                                                                                                  
refs/heads/production                                                                                               
refs/heads/sandbox/filippo/pontoon                                                                                  
refs/heads/sandbox/filippo/pontoon-o11y                                                                             
refs/heads/sandbox/kormat/pontoon-mariadb104-test                                                                   
refs/heads/wmcs/production

this gets translated to the following environments:

  • porduction - this is the production checkout using the sha-1 hash which is currently merge on production
  • dev_production - the production branch at head
  • dev_pontoon - the pontoon branch
  • sandbox_filippo_pontoon
  • sandbox_filippo_pontoon-o11y
  • sandbox_kormat_pontoon-mariadb104-test
  • wmcs_production

however this is not supper useful as most uses cant push branches to the operations/puppet repo. As such one is able to override the r10k sources and use any upstream they like. This allows users to fork operations/puppet and then set this as there source (with some modifications) giving ultimat freedome. As an example of how this may look i have created my own fork and for my use case i want to do the following things

  • ignore the real production - it has a few minor difference which cause issues
  • ignore all the pontoon and sandbox branches
  • create a new 'wmcs/' name space and map that to my environments. my upstream repo now has two branches
    • wmcs/production - a copy of production with minimal updates to make it work with environments
    • wmcs/puppetlabs - a branche where i want to play with the puppet labs puppet packages.

The config i have used is

role::puppetmaster::standalone::r10k_sources:
  b4ldr:
    ignore_branch_prefixes:
    - production
    - sandbox
    - pontoon
    remote: https://github.com/b4ldr/puppet-1
    strip_component: wmcs/

This means i end up with the following environments

There are a couple of caveats with this, specifically to use the strip_component options above one needs r10k >= 3.9 as well as a fix i have sent upstream. That said i have built a package that meets this requirement and we should be able to add it to the wikimedia-bulseye repo.