Page MenuHomePhabricator

puppetdb4: use postgres db backend in puppet-compiler
Closed, ResolvedPublic


The local database configuration deployed by the puppet_compiler puppet module is no longer working under puppetdb 4. According to the puppet docs the subprotocol setting used to specify hsqldb is now ignored by puppetdb and the only mentioned backend is postgres.

This is probably a good opportunity to move puppet compiler over to the role/profile paradigm as well. We could create a profile::puppet_compiler separate profile for a postgres backed puppet-compiler to deploy puppetdb4 compilers in parallel.

Event Timeline

herron triaged this task as Medium priority.Feb 13 2018, 9:37 PM
herron created this task.

Change 413881 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet_compiler: add support for puppetdb4 and local postgresql

While is still a work in progress, I was able to put together a postgres/puppetdb4 backed compiler instance using this patch with these steps:

  • Set up/use a self hosted puppetmaster to serve the patch above
  • Create a jessie instance with below puppet settings
  • In horizon "puppet configuration" for this instance...
    • apply class role::puppet_compiler
    • add hiera data:
etcd::peers_list: '%{::hostname}='
profile::puppetdb::master: <hostname of instance being created>
puppet_compiler::version: 0.4.0
puppet_major_version: 4
puppetdb_major_version: 4
- none
puppetdb::password::replication: none
puppetdb::password::rw: bar
realm: labs
  • Log in to the new instance
    • if using a small instance do something about the /srv filesystem - it's too small by default
    • append eqiad.wmnet codfw.wmnet esams.wmnet ulsfo.wmnet eqsin.wmnet to the dns search suffix list in /etc/resolv.conf (this is probably handled automatically somehow on compiler02/03?)
    • point the instance to your self hosted puppet master by editing /etc/puppet/puppet.conf. generate/sign puppet cert, etc. and then puppet agent -t
    • postgres, puppetdb, nginx and the compiler should be installed now
  • Copy facts from prod masters to the new compiler instance. There is a script to do this in operations/puppet which is meant to be run from a workstation (so the script can have ssh access to both prod and openstack). Substitute in the name of the compiler instance in the command below.
    • cd /path/to/git/operations/puppet
    • for MASTER in "puppetmaster1001 puppetmaster1002 puppetmaster2001 puppetmaster2002"; do PUPPET_MASTERS=${MASTER} PUPPET_COMPILER=<my_shiny_new_compiler_instance.domainname> modules/puppet_compiler/files/compiler-update-facts; done
  • Populate puppetdb - but there's a catch here - since the puppetdbquery module currently in production does not support puppetdb 4 a workaround is needed to populate the DB using an updated puppetdbquery. One approach is...
    • cp -a /var/lib/catalog-differ /var/lib/catalog-differ-puppetdb4
    • apply (puppetdbquery: upgrade to 3.0.1) to puppet code in /var/lib/catalog-differ-puppetdb4/production with something like cd /var/lib/catalog-differ-puppetdb4/production && git pull refs/changes/50/410050/4
    • populate puppetdb using the basedir containing the above patch /usr/local/bin/puppetdb-populate --basedir /var/lib/catalog-differ-puppetdb4/ --debug
  • Finally, run the compiler. But again because puppetdbquery isn't supporting puppetdb4 yet in production I'll use patch 410050 in the example (in this case only change will compile, while prod fails on puppetdbquery as expected)
    • CHANGE=410050 NODES=<insert_comma_delimited_list_of_fqdns_here> BUILD_NUMBER=<N> puppet-compiler --debug

Ready to begin upgrading the puppet compiler now (in preparation for the prod puppetdb upgrade). Here's the process I have in mind:

  1. disable jenkins worker compiler03.puppet3-diffs.eqiad.wmflabs of project operations-puppet-catalog-compiler - with the help of Release-Engineering-Team
  2. merge
  3. upgrade compiler03.puppet3-diffs.eqiad.wmflabs to local postgres/pdb4 backend
  4. sync facts and populate new puppetdb on compiler03.puppet3-diffs.eqiad.wmflabs
  5. production puppetdb upgrade happens (
  6. enable jenkins worker compiler03.puppet3-diffs.eqiad.wmflabs and disable worker compiler02.puppet3-diffs.eqiad.wmflabs - with the help of Release-Engineering-Team
  7. upgrade compiler02.puppet3-diffs.eqiad.wmflabs to local postgres/pdb4 backend
  8. enable jenkins worker compiler02.puppet3-diffs.eqiad.wmflabs - with the help of Release-Engineering-Team

Change 413881 merged by Herron:
[operations/puppet@production] puppet_compiler: add support for puppetdb4 and local postgresql

Compiler03 has been upgraded to puppetdb 4 and populate-puppetdb kicked off. The first few dozen hosts have compiled/populated successfully. Looks good. Going to leave the job running in a screen session to complete overnight and plan on picking up at step 5 ( in the morning.

compiler03 has been enabled and compiler02 disabled (in jenkins)

Running a test puppet compiler build works and looks good. Will proceed with upgrading compiler02 in the near future

herron claimed this task.

compiler02 has been upgraded and re-enabled which completes this task