There's already puppet compiler which lets you run it on a list of hosts. It's triggered via a Jenkins job.

That's https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/

If I understand correctly, that's triggered manually, and it runs on prod and not labs? Or, in any case, it has access to prod, so I can't use it ;-)

Setting it up for TL use sounds reasonable enough. Thanks for the suggestion!

Perhaps I should explain how I use Toolsbeta after I essentially usurped it :-).

I have "copied" Tools roles to Toolsbeta. As setting up new instances takes hours for the package installations to complete, they usually just idle, but at the moment there is no way for tests to start from scratch.

After creating a change on my box, I run puppet-test. This checks whether there are uncommitted changes. If not, it looks if ~/.puppet-test/$changeid.sh exists and sources it (think of it as a very small test "suite"). It executes puppet_test_pre.

Then it pushes to and checks out the change on toolsbeta-puppetmaster3. For this to work, you need to configure git per the instructions at "How do I submit my changes to gerrit?". After the check-out, it runs puppet agent -tv first on toolsbeta-puppetmaster3 and then on other hosts given by puppet_test_additional_hosts. Once all that is done, puppet_test_post is run.

To return to "normal", I ssh to toolsbeta-puppetmaster3, sudo, check out the production branch and git pull --rebase.

Depending on the complexity of the change and my gut feeling, I might only run puppet-test to see if a change applies and "looks alright", or I add toolsbeta-exec-01 or some other role instance to puppet_test_additional_hosts in ~/.puppet-test/$changeid.sh, run puppet-test and then check manually on the role instance whether a service is running/file has been created/deleted/etc., or in cases where reviewing might take longer and I'll have to revisit the issue in some weeks, I script sshing to the role instance in puppet_test_post so that I don't forget what I wanted to test.

I think puppet-test is thus almost what you mean by option 2. Now this obviously only works as long as only one developer is using Toolsbeta as a testing ground, and as running Puppet can take a while, using it for this purpose can block it for hours if you start a test, do something else while waiting, get back, rephrase the change, run another test, etc.

It could be enhanced to a more batch type processing, thus decreasing the run and "blocking" time: Get a "lock" on toolsbeta-puppetmaster3:/root/i-have-control, "reset" the hosts to HEAD^, run those steps described above while logging the output, "reset" the hosts to production, remove the "lock", present the user with the logs, optionally with an interactive phase in between.

If you think that could be a useful modus operandi, I would request a Git repository for puppet-test, and then we could work on improving it (and I wouldn't mind rewriting it in Python).

Thanks, that's a super useful explanation.

As I understand it, puppet-compiler should be able to give the relevant diffs (although not the actual diffs, but the compiled-manifest-to-compiled-manifest diffs -- but we have everything puppetized anyway, right ;-)), and I indeed think your puppet-test tool essentially does 2).

Adding locking (and maybe an irc ping 'hey, your change is fully deployed') would be a useful addition, though, although probably not critical in practice.

I'll play around with it, and then I'll try to write up a short document on 'developing, testing and deploying tool labs infra patches'

OK, so a bit more on puppet compiler. I tried adding a fake 'tools bastion host' to site.pp to run puppet-compiler on that:

node 'bastion.toollabs.compiler' {
    include role::labs::instance
    include role::labs::tools::bastion
}

but that gives me

[ 2015-11-19T20:10:25 ] ERROR: Unable to find facts for host bastion.toollabs.compiler, skipping

https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/1328/console#console-section-0

because (of course) puppet compiler doesn't know what Facter returns. https://wikitech.wikimedia.org/wiki/User:Giuseppe_Lavagetto/RefreshPuppetCompiler has notes on how puppet facts are imported, but that needs manual work. Still, if we can do that for a set of 'fake' tools hosts (because puppet compiler also needs them to be in site.pp, and we don't want the parameters there to actually be applied), then this should work!

...although it also needs to get Hiera data from somewhere, and that doesn't seem to be cached by the puppet master...

So: instead, setting up a puppet compiler instance I can login to myself (toolsbeta-valhallasw-puppet-compiler.toolsbeta.eqiad.wmflabs) to play around.

I'm abusing this task to jot down notes, as I tend to lose my paper notebooks ;-)

• toolsbeta-valhallasw-puppet-compiler is still alive
• /var/lib/puppet/yaml exists on both toolsbeta-puppetmaster3 and tools-puppetmaster-01. Have not checked labs puppetmaster.
• First trying toolsbeta-puppetmaster3. script seems to run correctly.
• scp toolsbeta-puppetmaster3.eqiad.wmflabs:puppet-facts.tar.xz . && scp puppet-facts.tar.xz toolsbeta-valhallasw-puppet-compiler.eqiad.wmflabs:. (direct scp a:x b:x failed??)
• cd /var/lib/catalog-differ/puppet/
• sudo sudo -u jenkins-deploy rm -rf yaml
• sudo sudo -u jenkins-deploy tar -xvJ < /home/valhallasw/puppet-facts.tar.xz
• CHANGE=296932 NODES=toolsbete-bastion BUILD_NUMBER=2 sudo -E sudo -Eu jenkins-deploy puppet-compiler

• error: copy-fd: write returned No space left on device because /mnt/jenkins-workspace is actually just a directory on the small /mnt mount.
  • sudo mv /mnt/jenkins-workspace/ /
  • sudo ln -s /jenkins-workspace/ /mnt/jenkins-workspace

ERROR: Unable to find facts for host toolsbeta-bastion, skipping

• CHANGE=296932 NODES=toolsbeta-bastion.toolsbeta.eqiad.wmflabs BUILD_NUMBER=5 sudo -E sudo -Eu jenkins-deploy puppet-compiler

[ 2016-07-01T19:08:06 ] INFO: Compiling host toolsbeta-bastion.toolsbeta.eqiad.wmflabs (production)
[ 2016-07-01T19:08:07 ] ERROR: Compilation failed for hostname toolsbeta-bastion.toolsbeta.eqiad.wmflabs  with the current tree.
[ 2016-07-01T19:08:07 ] INFO: Compilation exited with code 1

Error message is not actually mentioned in the output. Blergh. --debug parameter gives less output rather than more.

puppet-compiler code is in /var/lib/catalog-differ/compiler/, but that code does not seem to be directly used. Solved with sudo python setup.py develop

puppet master --vardir=/var/lib/catalog-differ/puppet --modulepath=/mnt/jenkins-workspace/puppet-compiler/1467401632/production/private/modules:/mnt/jenkins-workspace/puppet-compiler/1467401632/production/src/modules --confdir=/mnt/jenkins-workspace/puppet-compiler/1467401632/production/src --templatedir=/mnt/jenkins-workspace/puppet-compiler/1467401632/production/src/templates --compile=toolsbeta-bastion.toolsbeta.eqiad.wmflabs --color=false

shows the reason for failure is that I'm using sudo instead of being logged in as jenkins-deploy. Trying again with sudo sudo -iu jenkins-deploy bash..

Running

CHANGE=296932 NODES=toolsbeta-bastion.toolsbeta.eqiad.wmflabs BUILD_NUMBER=`date +%s`  puppet-compiler

again.

After fiddling with bind mounts, it seems to work! Sort of -- using https://gerrit.wikimedia.org/r/#/c/296932 I see no changes (because wikitech puppet class defs are not used?), and https://gerrit.wikimedia.org/r/#/c/254183 errors out due to a duplicate definition of /etc/ssh/ssh_known_hosts (Ssh::Client and Toollabs::init) --> puppet-compiler thinks we are in the production realm.

• puppet/yaml/node/toolsbeta-exec-101.toolsbeta.eqiad.wmflabs.yaml does specify the realm, but puppet/yaml/node/toolsbeta-bastion.toolsbeta.eqiad.wmflabs.yaml doesnt??

puppetVar:
  - "realm=labs"
  - "instancename=toolsbeta-exec-101"
  - "instanceproject=toolsbeta"
realm: labs

Adding that to toolsbeta-bastion.toolsbeta.eqiad.wmflabs.yaml doesn't work (gets removed??)

Let's use toolsbeta-exec-101.toolsbeta.eqiad.wmflabs.yaml as test host instead. Same issue: reference to realm gets removed from the node info...

Current hypothesis: puppet vars are all unset, and 'production' is default in realm.pp, so that's what is used.

Ok, so some progress: puppet-compiler at least works, and now I'm at the level where puppet needs to do the right thing. Not the most trivial part, but let's see if we can get anywhere.

using whatever was in /var/lib/git/ops/puppet originally as 'old' and the current production branch as 'current':

valhallasw@toolsbeta-puppetmaster3:~$ (cd /tmp/ops-puppet/ && git log -1) | head -n 1
commit 5feccac676e9066acb4b260760856eb0051f8832
valhallasw@toolsbeta-puppetmaster3:~$ (cd /var/lib/git/operations/puppet && git log -1) | head -n 1
commit cf2cbd82f51538cf812564e01c680fe9dc0dc3e7
valhallasw@toolsbeta-puppetmaster3:~$ sudo puppet master --compile toolsbeta-bastion.toolsbeta.eqiad.wmflabs > current
[...] 
valhallasw@toolsbeta-puppetmaster3:~$ sudo puppet master --modulepath=/etc/puppet/secret/modules:/etc/puppet/private/modules:/tmp/ops-puppet/modules --compile toolsbeta-bastion.toolsbeta.eqiad.wmflabs > old
[...]

then, after

puppet module install zack-catalog_diff
puppet catalog diff --show_resource_diff --content_diff old.pson current.pson

we find:

--------------------------------------------------------------------------------
current                                                       0.385767053526545%
--------------------------------------------------------------------------------
Added and removed resources:    +3 / 0
Params in new:
        File[/etc/cgconfig.d/utilities.conf]:
        content = checksumaec9cb0598cc9af94b0826ba739cdcadcontentgroup utilities {

    memory {

            memory.limit_in_bytes="2305843009213693951";

    }

    cpu {

            cpu.shares="512";

    }

}

Params in new:
        File[/etc/cgconfig.d/user-daemons.conf]:
        content = checksum192a9af699387b8a6eddb8695de45a40contentgroup user-daemons {

    memory {

            memory.limit_in_bytes="1152921504606846975";

    }

    cpu {

            cpu.shares="512";

    }

}
....etc

so that seems to work! Hopefully starting from there will help to figure out how to make puppet_compiler do the right thing on labs as well.

Failing hiera was because I forgot to set RUBYLIB=/mnt/jenkins-workspace/puppet-compiler/1467917450/production/src/modules/wmflib/lib/ , which meant the backends could not be found.

Hiera data is still not loaded, so copied hiera.yaml from toolsbeta-puppetmaster3's /etc/puppet/hiera.yaml. (= https://github.com/wikimedia/operations-puppet/blob/3218df65dcc4c9d42ce6deef0e130db817613f58/modules/puppetmaster/files/labs.hiera.yaml )

jenkins-deploy@toolsbeta-valhallasw-puppet-compiler:~$ cat > "/mnt/jenkins-workspace/puppet-compiler/1467917450/production/src/hiera.yaml"
:backends:
  - mwyaml
  - nuyaml
:nuyaml:
  :datadir: /etc/puppet/hieradata
:mwyaml:
  :host: https://wikitech.wikimedia.org
  :cache_ttl: 120
:private:
  :datadir: /etc/puppet/private/hieradata
:secret:
  :datadir: /etc/puppet/secret/hieradata
:hierarchy:
  - "labs/hosts/%{::hostname}"
  - "labs/%{::labsproject}/host/%{::hostname}"
  - "labs/%{::labsproject}/common"
  - "labs"
  - "secret/%{::labsproject}"
  - "private/%{::labsproject}"
  - common
  - "secret/common"
  - "private/common"

but that has hardcoded paths, so changed to

:nuyaml:
  :datadir: /mnt/jenkins-workspace/puppet-compiler/1467917450/production/src/hieradata
:mwyaml:
  :host: https://wikitech.wikimedia.org
  :cache_ttl: 120
:private:
  :datadir: /mnt/jenkins-workspace/puppet-compiler/1467917450/production/private/hieradata

Of course, I could have known that if I had read the puppet_compiler source -- https://github.com/wikimedia/operations-software-puppet-compiler/blob/569c30d3df9f9d21fd85942a57e5b10d9c864a31/puppet_compiler/prepare.py#L102

The node configuration is still failing -- the ldap node classifier is not being used.

Next hack: copy /etc/puppet/puppet.conf from puppetmaster3 to /mnt/jenkins-workspace/puppet-compiler/1467917450/production/src/puppet.conf (which, according to strace, is read by puppet master)

That is indeed read, but fails on all kinds of paths. Let's just take the node classification stuff out, and only keep the ldap lines...

and the manifest is compiled correctly!

So, in the end, three things are required:

• labs facts need to be available,
• prepare.py needs to be adapted for labs usage, using the labs hiera config rather than the prod one,
• prepare.py should create a file x/src/puppet.conf that contains the ldap credentials

I will try to create a patch for 2) and 3), and then hopefully we can also figure out a way to do 1).

Change 297902 had a related patch set uploaded (by Merlijn van Deen):
Set up labs realm (ldap classifier and hiera)

https://gerrit.wikimedia.org/r/297902

Change 302683 had a related patch set uploaded (by Merlijn van Deen):
puppet_compiler: add packages for labs realm

https://gerrit.wikimedia.org/r/302683

Change 302683 merged by Alexandros Kosiaris:
puppet_compiler: add packages for labs realm

https://gerrit.wikimedia.org/r/302683

Change 297902 merged by jenkins-bot:
Set up labs realm (ldap classifier and hiera)

https://gerrit.wikimedia.org/r/297902

Steps to setting up the compiler for tools:

1. create a new m1.large instance in toolsbeta (toolsbeta-valhallasw-puppet-compiler-5) (Jessie, don't forget the webserver security group)
2. set the following hiera config:

discovery::app_routes: {}
etcd::peers_list: []
labsdnsconfig: {}

3. apply role puppet_compiler
4. build
5. run PUPPET_MASTER=tools-puppetmaster-02.tools.eqiad.wmflabs PUPPET_COMPILER=toolsbeta-valhallasw-puppet-compiler-3.toolsbeta.eqiad.wmflabs modules/puppet_compiler/files/compiler-update-facts locally
6. ssh tools-puppetmaster-02.tools.eqiad.wmflabs
7. sudo su - jenkins-deploy
8. CHANGE=296932 NODES=tools-bastion-03.tools.eqiad.wmflabs BUILD_NUMBER=6 puppet-compiler
9. create proxy entry for http://tools-puppet-compiler.wmflabs.org/

I think the main question here is whether we want a seperate compiler for labs/tool labs, or whether we want to merge it with the existing one. I think the former is probably more practical, as it allows tools admins to keep track of their own fact updates, without running the risk of interfering with the one used for prod.

@hashar: how much work would it be to create an extra puppet-compiler job in Jenkins for this?

CMIIW, but is the compiler really working? http://tools-puppet-compiler.wmflabs.org/324623000/ says "Hosts that have no differences" includes "tools-grid-master.tools.eqiad.wmflabs/", but on a host with the class toollabs::master the file /var/lib/gridengine/default/common/host_aliases should be changed by that patch (https://gerrit.wikimedia.org/r/#/c/324623/).

@hashar: how much work would it be to create an extra puppet-compiler job in Jenkins for this?

I am not sure. The Jenkins job is not yet in the configuration management system for JJB (T97513), though that is fairly trivial to write in JJB, the job just:

• Ask parameter GERRIT_CHANGE_NUMBER
• Ask parameter LIST_OF_NODES
• Run CHANGE="${GERRIT_CHANGE_NUMBER}" NODES="${LIST_OF_NODES}" puppet-compiler

The instance in tools labs will need:

security

Allow ssh connection (port 22) from the Jenkins master (contint1001.wikimedia.org 208.80.154.17). The security rule can be added via Horizon.

Add labs user jenkins-deploy to the project.

Basic puppet provisioning

Apply the puppet role role::ci::slave::labs::common modules/role/manifests/ci/slave/labs/common.pp. This might well clash/conflict with other tools labs or the puppet-compiler classes. Roughly it does:

• Mount extra disk space to /mnt (though CI has a cherry pick to use /srv).
• Create a homedir for jenkins-deploy user
• Install openjdk-7-jre-headless

Then we can add the instance as a slave at https://integration.wikimedia.org/ci/computer/

In T97081#2840111, @scfc wrote:

CMIIW, but is the compiler really working? http://tools-puppet-compiler.wmflabs.org/324623000/ says "Hosts that have no differences" includes "tools-grid-master.tools.eqiad.wmflabs/", but on a host with the class toollabs::master the file /var/lib/gridengine/default/common/host_aliases should be changed by that patch (https://gerrit.wikimedia.org/r/#/c/324623/).

No, you're completely right -- I had not noticed that. The rabbit hole was quite deep:

• the installed branch is 0.0.4 by default, and my changes are in a much more recent version of puppet-compiler (fixable by setting puppet_compiler::version: '0.1.6', or whatever the relevant version will be, in Hiera)
• my changes use the ldap classifier, which isn't used anymore,
• the code was slightly refactored at some point, which meant the 'labs' realm was no longer set correctly, (both https://gerrit.wikimedia.org/r/325042)
• this needs puppet-enc installed on the host. I did this manually now, but there's also two patches in gerrit (https://gerrit.wikimedia.org/r/#/c/325046/, https://gerrit.wikimedia.org/r/#/c/325053/)
• there was some code in private hiera/secrets but not yet in the not-private-private-repo (https://gerrit.wikimedia.org/r/325041, https://gerrit.wikimedia.org/r/325050)

so... I have rewritten the code to use the enc classifier. The host now compiles correctly: http://tools-puppet-compiler.wmflabs.org/324623004/tools-grid-master.tools.eqiad.wmflabs/prod.tools-grid-master.tools.eqiad.wmflabs.pson
but... the change is in a file, and not in the manifest, so the compiler doesn't spot the difference ;-)

However, https://gerrit.wikimedia.org/r/#/c/324957/ now compiles sensibly to http://tools-puppet-compiler.wmflabs.org/324957001/ :-)

@hashar: Ok, that sounds doable! I'm first going to make sure these patches get through, and then I'll get back to you. I think puppet_compiler might actually already include that class, as the /mnt dir, jenkins-deploy user and openjdk install are already there. What credentials does the master use to connect (there is no pubkey in /mnt/home/jenkins-deploy). Anyway, now it's time for bed!

That is an impressive number of patches Merlijn :)

What credentials does the master use to connect (there is no pubkey in /mnt/home/jenkins-deploy).

The jenkins-deploy user is in LDAP and has a public key attached. That lets the Jenkins master magically connect!

I'm trying to reproduce the tools puppet compiler described here. A few things have clearly changed since this was last built... The hiera setup I seem to need looks like this:

discovery::app_routes: {}
etcd::cluster_name: toolsbeta-etcd-andrew
etcd::cluster_state: new
etcd::host: '%{::fqdn}'
etcd::peers_list: toolsbeta-andrew-puppet-compiler=http://toolsbeta-andrew-puppet-compiler.toolsbeta.eqiad.wmflabs:2380
etcd::port: 2739
etcd::ssl::puppet_cert_name: '%{::fqdn}'
etcd::use_ssl: true
labsdnsconfig: {}

The next stumbling block is:

5. run PUPPET_MASTER=tools-puppetmaster-02.tools.eqiad.wmflabs PUPPET_COMPILER=toolsbeta-valhallasw-puppet-compiler-3.toolsbeta.eqiad.wmflabs modules/puppet_compiler/files/compiler-update-facts locally

• Am I running this as root or as myself?

• Clearly that only works with the proper cwd; I'm guessing it should be /var/lib/catalog-differ/production.

• When I run the above command in that directory the script attempts to ssh to the tools puppetmaster, which fails. Is that expected? If so, what key should it be using?

In T97081#3665495, @Andrew wrote:

5. run PUPPET_MASTER=tools-puppetmaster-02.tools.eqiad.wmflabs PUPPET_COMPILER=toolsbeta-valhallasw-puppet-compiler-3.toolsbeta.eqiad.wmflabs modules/puppet_compiler/files/compiler-update-facts locally

• Am I running this as root or as myself?

I believe as jenkins-deploy user, i.e.

user@host$ sudo sudo -iu jenkins-deploy bash
jenkins-deploy@host$ CHANGE=296932 NODES=toolsbeta-bastion.toolsbeta.eqiad.wmflabs BUILD_NUMBER=`date +%s`  puppet-compiler

• Clearly that only works with the proper cwd; I'm guessing it should be /var/lib/catalog-differ/production.

As I remember it the cwd did not matter -- the puppet-compiler command uses hardcoded paths. But the command may have changed over time, of course...

• When I run the above command in that directory the script attempts to ssh to the tools puppetmaster, which fails. Is that expected? If so, what key should it be using?

Also not sure -- I don't remember this happening. Maybe the update-facts command is now automatically run when a new manifest is compiled?

In T97081#2843075, @valhallasw wrote:

• the code was slightly refactored at some point, which meant the 'labs' realm was no longer set correctly, (both https://gerrit.wikimedia.org/r/325042)
• this needs puppet-enc installed on the host. I did this manually now, but there's also two patches in gerrit (https://gerrit.wikimedia.org/r/#/c/325046/, https://gerrit.wikimedia.org/r/#/c/325053/)

Note that these changes have not been merged, so some of these changes might need to be applied locally to get everything working again.

Change 383857 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] compiler-update-facts: restore optional use of PUPPET_MASTER env

https://gerrit.wikimedia.org/r/383857

Here's my latest attempt to describe what works. Once the concerned patches are merged I'll try to get this down on wikitech someplace.

How to build a puppet compiler node <worker> to test instance <instance> managed by puppetmaster <master>

• Build <worker> as a large Jessie instance (probably anywhere, but ideally in the same project as <master> if <master> is a VM
• On <worker>, apply the classes "role::puppet_compiler" and "openstack::puppet::master::enc"
• On <worker>, apply the following hiera:

discovery::app_routes: {}
etcd::cluster_name: <something arbitrary and distinctive>
etcd::cluster_state: new
etcd::host: '%{::fqdn}'
etcd::peers_list: <worker hostname>=http://<worker fqdn>:2380
etcd::port: 2739
etcd::ssl::puppet_cert_name: '%{::fqdn}'
etcd::use_ssl: true
labsdnsconfig: {}
openstack::puppet::master::enc::puppetmaster labs-pupppetmaster.wikimedia.org

• On a local (not a VM) with keys installed to access all the necessary hosts, run this script from the puppet repo:

$ PUPPET_MASTERS=<master> PUPPET_COMPILER=<worker> modules/puppet_compiler/files/compiler-update-facts

• Set up a web proxy (ideally named puppet-compiler.wmflabs.org) pointing to <worker>
• Make sure port 80 is open on <worker> so the proxy can reach it

This assumes that https://gerrit.wikimedia.org/r/#/c/383857 is merged. If not, that local fix needs to be applied to compiler-update-facts.

These instructions presume that https://gerrit.wikimedia.org/r/#/c/325042/ is merged. If it isn't, that patch has to be applied by hand
on <worker> in /var/lib/catalog-differ/compiler, and puppet disabled,and then the change actually installed by running

# python setup.py develop

in /var/lib/catalog-differ/compiler

Once all that is done, you can invoke a compiler test on <worker> like this:

sudo su - jenkins-deploy
CHANGE=<patch of interest> NODES=<node of interest> BUILD_NUMBER=8 puppet-compiler

Change 383857 merged by Andrew Bogott:
[operations/puppet@production] compiler-update-facts: restore optional use of PUPPET_MASTER env

https://gerrit.wikimedia.org/r/383857