Change 413881 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] puppet_compiler: add support for puppetdb4 and local postgresql

https://gerrit.wikimedia.org/r/413881

While https://gerrit.wikimedia.org/r/413881 is still a work in progress, I was able to put together a postgres/puppetdb4 backed compiler instance using this patch with these steps:

• Set up/use a self hosted puppetmaster to serve the patch above
• Create a jessie instance with below puppet settings
• In horizon "puppet configuration" for this instance...
  • apply class role::puppet_compiler
  • add hiera data:

etcd::peers_list: '%{::hostname}=http://127.0.0.1:2380'
profile::puppetdb::master: <hostname of instance being created>
puppet_compiler::version: 0.4.0
puppet_major_version: 4
puppetdb_major_version: 4
profile::puppetdb::slaves:
- none
puppetdb::password::replication: none
puppetdb::password::rw: bar
realm: labs

• Log in to the new instance
  • if using a small instance do something about the /srv filesystem - it's too small by default
  • append wikimedia.org eqiad.wmnet codfw.wmnet esams.wmnet ulsfo.wmnet eqsin.wmnet to the dns search suffix list in /etc/resolv.conf (this is probably handled automatically somehow on compiler02/03?)
  • point the instance to your self hosted puppet master by editing /etc/puppet/puppet.conf. generate/sign puppet cert, etc. and then puppet agent -t
  • postgres, puppetdb, nginx and the compiler should be installed now
• Copy facts from prod masters to the new compiler instance. There is a script to do this in operations/puppet which is meant to be run from a workstation (so the script can have ssh access to both prod and openstack). Substitute in the name of the compiler instance in the command below.
  • cd /path/to/git/operations/puppet
  • for MASTER in "puppetmaster1001 puppetmaster1002 puppetmaster2001 puppetmaster2002"; do PUPPET_MASTERS=${MASTER} PUPPET_COMPILER=<my_shiny_new_compiler_instance.domainname> modules/puppet_compiler/files/compiler-update-facts; done

• Populate puppetdb - but there's a catch here - since the puppetdbquery module currently in production does not support puppetdb 4 a workaround is needed to populate the DB using an updated puppetdbquery. One approach is...
  • cp -a /var/lib/catalog-differ /var/lib/catalog-differ-puppetdb4
  • apply https://gerrit.wikimedia.org/r/#/c/410050/ (puppetdbquery: upgrade to 3.0.1) to puppet code in /var/lib/catalog-differ-puppetdb4/production with something like cd /var/lib/catalog-differ-puppetdb4/production && git pull https://gerrit.wikimedia.org/r/operations/puppet refs/changes/50/410050/4
  • populate puppetdb using the basedir containing the above patch /usr/local/bin/puppetdb-populate --basedir /var/lib/catalog-differ-puppetdb4/ --debug
• Finally, run the compiler. But again because puppetdbquery isn't supporting puppetdb4 yet in production I'll use patch 410050 in the example (in this case only change will compile, while prod fails on puppetdbquery as expected)
  • CHANGE=410050 NODES=<insert_comma_delimited_list_of_fqdns_here> BUILD_NUMBER=<N> puppet-compiler --debug

Ready to begin upgrading the puppet compiler now (in preparation for the prod puppetdb upgrade). Here's the process I have in mind:

1. disable jenkins worker compiler03.puppet3-diffs.eqiad.wmflabs of project operations-puppet-catalog-compiler - with the help of Release-Engineering-Team
2. merge https://gerrit.wikimedia.org/r/413881
3. upgrade compiler03.puppet3-diffs.eqiad.wmflabs to local postgres/pdb4 backend
4. sync facts and populate new puppetdb on compiler03.puppet3-diffs.eqiad.wmflabs
5. production puppetdb upgrade happens (https://phabricator.wikimedia.org/T177253#4056981)
6. enable jenkins worker compiler03.puppet3-diffs.eqiad.wmflabs and disable worker compiler02.puppet3-diffs.eqiad.wmflabs - with the help of Release-Engineering-Team
7. upgrade compiler02.puppet3-diffs.eqiad.wmflabs to local postgres/pdb4 backend
8. enable jenkins worker compiler02.puppet3-diffs.eqiad.wmflabs - with the help of Release-Engineering-Team

Mentioned in SAL (#wikimedia-releng) [2018-03-19T16:11:58Z] <bd808> jenkins: compiler03.puppet3-diffs offline for https://phabricator.wikimedia.org/T187258

Change 413881 merged by Herron:
[operations/puppet@production] puppet_compiler: add support for puppetdb4 and local postgresql

https://gerrit.wikimedia.org/r/413881

Compiler03 has been upgraded to puppetdb 4 and populate-puppetdb kicked off. The first few dozen hosts have compiled/populated successfully. Looks good. Going to leave the job running in a screen session to complete overnight and plan on picking up at step 5 (https://phabricator.wikimedia.org/T187258#4061133) in the morning.

compiler03 has been enabled and compiler02 disabled (in jenkins)

Running a test puppet compiler build works and looks good. Will proceed with upgrading compiler02 in the near future