Page MenuHomePhabricator

Support targetting WMCS instances with the Jenkins puppet compiler
Closed, ResolvedPublic

Description

I have good proof-of-concept patches for this, but there are some open questions.

  • Does it work to add .eqiad.wmflabs facts to the existing clients? It does.
  • Won't this break puppetdb-populate? It does not.
  • Some cloud puppetmasters (including the central one) don't have puppetdb. This is addressed by https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/499007/ but we could adopt a more clever approach so the update-facts script doesn't have to know where puppetdb is or isn't.
  • The wildcard "" hostname should probably only include prod hosts and exclude wmflabs hosts; otherwise, wildcard runs will take FOREVER
  • We need to a way to enumerate fact sources. Currently we do this with a clever hiera pull for the production puppetmasters. @herron suggests we just replace this with a simple flat file for everything -- resolved by specifying puppetmaster in env
  • Adding puppetmasters from other environments to the existing sync script model means users of the sync script will need access to all the puppet masters involved, or some way to exclude/select master. Otherwise the script will fail, and the number of people able to sync facts will be few -- resolved with patchsets that store per-puppetmaster facts

Event Timeline

Change 499007 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] puppet-compiler: restore the ability to export facts without puppetdb

https://gerrit.wikimedia.org/r/499007

Change 499026 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] puppet compiler: collect facts from cloud VMs as well as prod hosts

https://gerrit.wikimedia.org/r/499026

Andrew renamed this task from Allow us to target VPS servers on the Jenkins puppet compiler to Support targetting WMCS instances with the Jenkins puppet compiler.Mar 27 2019, 6:15 PM

The compiler checks subdirs of /var/lib/catalog-differ/puppet/yaml/facts. That means that we can potentially have two different compiler-update-facts scripts, one that uploads prod facts (into prod/) and one that uploads cloud facts (into cloud/). That solves the issue with keys and access, if we determine that that's an actual problem.

Change 500501 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] compiler-update-facts: better support addition of arbitrary fact sets

https://gerrit.wikimedia.org/r/500501

Change 499026 abandoned by Andrew Bogott:
puppet compiler: collect facts from cloud VMs as well as prod hosts

Reason:
Abandoned in favor of https://gerrit.wikimedia.org/r/#/c/operations/puppet/ /500501/

https://gerrit.wikimedia.org/r/499026

Change 501039 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/software/puppet-compiler@master] utils.facts_file: do a recursive search in the 'facts' dir

https://gerrit.wikimedia.org/r/501039

Change 499007 merged by Andrew Bogott:
[operations/puppet@production] puppet-compiler: restore the ability to export facts without puppetdb

https://gerrit.wikimedia.org/r/499007

Change 501039 merged by Andrew Bogott:
[operations/software/puppet-compiler@master] support multiple facts dirs

https://gerrit.wikimedia.org/r/501039

Change 500501 merged by Andrew Bogott:
[operations/puppet@production] compiler-update-facts: better support addition of arbitrary fact sets

https://gerrit.wikimedia.org/r/500501

Change 502026 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/software/puppet-compiler@master] Bump setup.py version to 0.5.0

https://gerrit.wikimedia.org/r/502026

Change 502026 merged by Andrew Bogott:
[operations/software/puppet-compiler@master] Bump setup.py version to 0.5.0

https://gerrit.wikimedia.org/r/502026

Change 502028 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] puppet-compiler: install latest version

https://gerrit.wikimedia.org/r/502028

Change 502028 merged by Andrew Bogott:
[operations/puppet@production] puppet-compiler: install latest version

https://gerrit.wikimedia.org/r/502028

Mentioned in SAL (#wikimedia-cloud) [2019-04-07T22:18:14Z] <andrewbogott> upgrade puppet-compiler version to 0.5.0 (via hiera setting on Horizon) for T219430

This pretty much works. A few things that need further consideration:

  • Given that we have regex processing, is the "" use case worth fixing, or should we just rip it out?
  • The mtime-comparison system for comparing fact freshness may or may not be correct. Does it compare freshness of the fact files being generated on the puppetmasters, or just the most recently updated via compiler-update-facts? If the latter, could compiler-update-facts preserve mtimes from the puppetmasters?

I updated the docs. The "" use case seems to be working fine, so I think this can be closed barring anyone finding issues.