puppetdb4: use postgres db backend in puppet-compiler
Closed, ResolvedPublic

Description

The local database configuration deployed by the puppet_compiler puppet module is no longer working under puppetdb 4. According to the puppet docs the subprotocol setting used to specify hsqldb is now ignored by puppetdb and the only mentioned backend is postgres. https://puppet.com/docs/puppetdb/4.4/configure.html#subprotocol

This is probably a good opportunity to move puppet compiler over to the role/profile paradigm as well. We could create a profile::puppet_compiler separate profile for a postgres backed puppet-compiler to deploy puppetdb4 compilers in parallel.

herron created this task.Feb 13 2018, 9:37 PM
herron triaged this task as Normal priority.

Change 413881 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet_compiler: add support for puppetdb4 and local postgresql

https://gerrit.wikimedia.org/r/413881

While https://gerrit.wikimedia.org/r/413881 is still a work in progress, I was able to put together a postgres/puppetdb4 backed compiler instance using this patch with these steps:

  • Set up/use a self hosted puppetmaster to serve the patch above
  • Create a jessie instance with below puppet settings
  • In horizon "puppet configuration" for this instance...
    • apply class role::puppet_compiler
    • add hiera data:
etcd::peers_list: '%{::hostname}=http://127.0.0.1:2380'
profile::puppetdb::master: <hostname of instance being created>
puppet_compiler::version: 0.4.0
puppet_major_version: 4
puppetdb_major_version: 4
profile::puppetdb::slaves:
- none
puppetdb::password::replication: none
puppetdb::password::rw: bar
realm: labs
  • Log in to the new instance
    • if using a small instance do something about the /srv filesystem - it's too small by default
    • append wikimedia.org eqiad.wmnet codfw.wmnet esams.wmnet ulsfo.wmnet eqsin.wmnet to the dns search suffix list in /etc/resolv.conf (this is probably handled automatically somehow on compiler02/03?)
    • point the instance to your self hosted puppet master by editing /etc/puppet/puppet.conf. generate/sign puppet cert, etc. and then puppet agent -t
    • postgres, puppetdb, nginx and the compiler should be installed now
  • Copy facts from prod masters to the new compiler instance. There is a script to do this in operations/puppet which is meant to be run from a workstation (so the script can have ssh access to both prod and openstack). Substitute in the name of the compiler instance in the command below.
    • cd /path/to/git/operations/puppet
    • for MASTER in "puppetmaster1001 puppetmaster1002 puppetmaster2001 puppetmaster2002"; do PUPPET_MASTERS=${MASTER} PUPPET_COMPILER=<my_shiny_new_compiler_instance.domainname> modules/puppet_compiler/files/compiler-update-facts; done
  • Populate puppetdb - but there's a catch here - since the puppetdbquery module currently in production does not support puppetdb 4 a workaround is needed to populate the DB using an updated puppetdbquery. One approach is...
    • cp -a /var/lib/catalog-differ /var/lib/catalog-differ-puppetdb4
    • apply https://gerrit.wikimedia.org/r/#/c/410050/ (puppetdbquery: upgrade to 3.0.1) to puppet code in /var/lib/catalog-differ-puppetdb4/production with something like cd /var/lib/catalog-differ-puppetdb4/production && git pull https://gerrit.wikimedia.org/r/operations/puppet refs/changes/50/410050/4
    • populate puppetdb using the basedir containing the above patch /usr/local/bin/puppetdb-populate --basedir /var/lib/catalog-differ-puppetdb4/ --debug
  • Finally, run the compiler. But again because puppetdbquery isn't supporting puppetdb4 yet in production I'll use patch 410050 in the example (in this case only change will compile, while prod fails on puppetdbquery as expected)
    • CHANGE=410050 NODES=<insert_comma_delimited_list_of_fqdns_here> BUILD_NUMBER=<N> puppet-compiler --debug

Ready to begin upgrading the puppet compiler now (in preparation for the prod puppetdb upgrade). Here's the process I have in mind:

  1. disable jenkins worker compiler03.puppet3-diffs.eqiad.wmflabs of project operations-puppet-catalog-compiler - with the help of Release-Engineering-Team
  2. merge https://gerrit.wikimedia.org/r/413881
  3. upgrade compiler03.puppet3-diffs.eqiad.wmflabs to local postgres/pdb4 backend
  4. sync facts and populate new puppetdb on compiler03.puppet3-diffs.eqiad.wmflabs
  5. production puppetdb upgrade happens (https://phabricator.wikimedia.org/T177253#4056981)
  6. enable jenkins worker compiler03.puppet3-diffs.eqiad.wmflabs and disable worker compiler02.puppet3-diffs.eqiad.wmflabs - with the help of Release-Engineering-Team
  7. upgrade compiler02.puppet3-diffs.eqiad.wmflabs to local postgres/pdb4 backend
  8. enable jenkins worker compiler02.puppet3-diffs.eqiad.wmflabs - with the help of Release-Engineering-Team

Mentioned in SAL (#wikimedia-releng) [2018-03-19T16:11:58Z] <bd808> jenkins: compiler03.puppet3-diffs offline for https://phabricator.wikimedia.org/T187258

Change 413881 merged by Herron:
[operations/puppet@production] puppet_compiler: add support for puppetdb4 and local postgresql

https://gerrit.wikimedia.org/r/413881

herron added a comment.EditedMar 19 2018, 8:35 PM

Compiler03 has been upgraded to puppetdb 4 and populate-puppetdb kicked off. The first few dozen hosts have compiled/populated successfully. Looks good. Going to leave the job running in a screen session to complete overnight and plan on picking up at step 5 (https://phabricator.wikimedia.org/T187258#4061133) in the morning.

compiler03 has been enabled and compiler02 disabled (in jenkins)

Running a test puppet compiler build works and looks good. Will proceed with upgrading compiler02 in the near future

herron closed this task as Resolved.Mar 29 2018, 4:41 PM
herron claimed this task.

compiler02 has been upgraded and re-enabled which completes this task