Change 179244 merged by jenkins-bot:
Job operations-puppet-bundle-rspec

https://gerrit.wikimedia.org/r/179244

Poked our internal ops list to attract attention on the patch and have people to try it out.

Change 178810 abandoned by Hashar:
Basic rspec setup

Reason:
Dan Duvall is working on a similar rspec suite for mediawiki/vagrant. Once we acquired experience there, it would make it easier to reuse the same model for operations/puppet.git

The idea is tracked by T78342 so there is little point in keep the Gerrit change opened.

https://gerrit.wikimedia.org/r/178810

There was some rspec incompatibility issue for which I proposed a fix at https://github.com/rodjek/rspec-puppet/pull/243 which is superseded by https://github.com/rodjek/rspec-puppet/pull/283

The job used to run those rspec was disabled recently in https://gerrit.wikimedia.org/r/#/c/179244/. The issue was that different modules used different dependencies / versions, which makes it hard to run everything from a central Rakefile. The goal is to run all tests with a single bundler execution.

We gave it a try a while ago. @Gehel proposed to have some kind of hacking sprint to revisit running the Puppet module tests easily.

At first, for CI purposes we are relying on http://bundler.io/ which given a Gemfile setup an appropriate environment to run commands into. A typical example would be:

bundle install
bundle exec rake -T  # list rake tasks

A very long time ago, in puppet /Rakefile I have created a spec task which glob('modules/*/Rakefile') to detect modules potentially have tests then invoke rake spec_standalone inside the module (see function run_module_spec). The rake task collect passing/failling modules and report back.

One can give it a try with:

bundle install
bundle exec spec

Which eventually leads to:

--------------------------------------------------------------------------------
Finished running tests for all modules
--------------------------------------------------------------------------------

The following modules are NOT passing tests:
- backup
- bacula
- base
- elasticsearch
- git
- install_server
- mirrors
- mysql
- nginx
- nrpe
- osm
- postgresql
- rsync
- servermon
- service
- squid3
- stdlib
- strongswan
- wmflib

Some example errors:

LoadError: cannot load such file -- puppetlabs_spec_helper/rake_tasks
Don't know how to build task 'spec_standalone'
Errno::ENOTDIR: Not a directory @ dir_s_rmdir - spec/fixtures/modules

Some modules use puppetlabs_spec_helper/rake_tasks which provides the spec_standalone task. Some other modules just use rspec. So one will have to figure out the proper entry point to invoke.

Then IIRC modules have different conflicting dependencies. Some would want rspec 2.14, others rspec 2.99 and so on.

Potentially, at the root of puppet.git, the Rakefile would glob modules having Rakefile then for each of them:

cd modules/foo
bundle install
bundle exec rake puppet_spec

The /Gemfile would just have the dependencies needed to invoke that rake task. Each modules would in turn have the Rakefile to define the puppet_spec task which would invoke either spec or spec_standalone depending on the test system the module rely on. Additionally, each modules would need a Gemfile listing its dependencies, so one can use whatever rspec or puppetlabs_spec_helper version.

There might be a better approach :-)

A change that got abandoned which might be of interest https://gerrit.wikimedia.org/r/#/c/178810/ . That one is to run unit tests for the various ruby files we have in the wmflibs module. Feel free to have it reopened / take over :-}

Change 178810 restored by Hashar:
Basic rspec setup

https://gerrit.wikimedia.org/r/178810

I have reopened/rebased/fixed https://gerrit.wikimedia.org/r/#/c/178810/ which introduce a very basic spec for wmflib.os_version(). Can be invoked with:

bundle install
cd modules/wmflib/
bundle exec rspec

That is using rspec 3.x. Some other spec in wmflib modules are written for rspec 2.x and use the should syntax which is deprecated. That is an example of each modules having different version requirements :-(

Change 282461 had a related patch set uploaded (by Nicko):
T78342 excluding spec folders in .gitignore file

https://gerrit.wikimedia.org/r/282461

Sorry for being so dumb, but I had a hard time finding how to execute bundle exec spec, it was actually because the command to execute is bundle exec rake spec (ok maybe it won't help anyone but just in case)

Change 282461 abandoned by Nicko:
T78342 excluding spec folders in .gitignore file

Reason:
I messed up with git basically

https://gerrit.wikimedia.org/r/282461

Change 282484 had a related patch set uploaded (by Nicko):
Modification of Rakefile

https://gerrit.wikimedia.org/r/282484

Change 296821 had a related patch set uploaded (by Hashar):
network: simplify tests with puppetlabs_spec_helper

https://gerrit.wikimedia.org/r/296821

@akosiaris recently introduced the network puppet module and we have something nice going on with rspec-puppet and puppetlabs_spec_helper gems.

https://gerrit.wikimedia.org/r/#/c/296821/ is a working proof of concept showing the basic glue is quite trivial once one has figured it out. We can then generalize to the rest of the modules and follow up on Nicko patch to have a single entry point at the root of the repo.

Change 296821 merged by Alexandros Kosiaris:
network: simplify tests with puppetlabs_spec_helper

https://gerrit.wikimedia.org/r/296821

Change 296927 had a related patch set uploaded (by Hashar):
rsync: fix spec that used ruby 1.8 syntax

https://gerrit.wikimedia.org/r/296927

Change 296927 merged by Alexandros Kosiaris:
rsync: fix spec that used ruby 1.8 syntax

https://gerrit.wikimedia.org/r/296927

Change 296933 had a related patch set uploaded (by Hashar):
mysql: fix spec

https://gerrit.wikimedia.org/r/296933

Change 296965 had a related patch set uploaded (by Hashar):
(WIP) rspec config for the whole repository (WIP)

https://gerrit.wikimedia.org/r/296965

https://gerrit.wikimedia.org/r/296965 is an attempt to let us run rspec from the root of the repo. That terribly fails though:

749 examples, 341 failures, 2 pending

Change 297129 had a related patch set uploaded (by Hashar):
scap: align spec with other modules

https://gerrit.wikimedia.org/r/297129

Change 296933 merged by Alexandros Kosiaris:
mysql: fix spec

https://gerrit.wikimedia.org/r/296933

Change 297315 had a related patch set uploaded (by Hashar):
nrpe: fix spec to use hiera

https://gerrit.wikimedia.org/r/297315

Change 297315 merged by Alexandros Kosiaris:
nrpe: fix spec to use hiera

https://gerrit.wikimedia.org/r/297315

Change 297129 merged by Alexandros Kosiaris:
scap: align spec with other modules

https://gerrit.wikimedia.org/r/297129

Change 297370 had a related patch set uploaded (by Hashar):
git: align spec configuration

https://gerrit.wikimedia.org/r/297370

Change 297378 had a related patch set uploaded (by Hashar):
spec fix for aptrepo and installserver

https://gerrit.wikimedia.org/r/297378

Change 297370 merged by Gehel:
git: align spec configuration

https://gerrit.wikimedia.org/r/297370

Change 296965 abandoned by Hashar:
(WIP) rspec config for the whole repository (WIP)

Reason:
After some hassle, there is not much we can do to be able to run rspec from the root of the repo. So will go down to use something like:

find . -path './module/*/spec' -execdir bundle exec rake spec

https://gerrit.wikimedia.org/r/296965

Thanks to @Nicko we have a good dynamic now. The logic is roughly:

• Rakefile with require puppetslabs_spec_helper/rake_tasks
• a .fixtures.yaml file that list the module dependencies
• bundle exec rake spec which runs the tasks:
  • spec_prep to bring up the fixtures
  • spec_standalone run tests
  • spec_clean gc the fixtures if tests passed

The patch https://gerrit.wikimedia.org/r/#/c/282484/ introduces rake spec at the root of the repo which lamely run bundle exec rake spec under each module having a spec directory.

Some modules still needs to be adjusted, @akosiaris and @Gehel have been lightning fast at reviewing and approving the spam of changes I have sent since Friday.

Guess we will want some basic documentation, maybe on wikitech as to how to run tests then craft a Jenkins job that would run the spec whenever a patch is proposed on puppet.

From there:

• get moar spec
• produce coverage
• run in parallel to speed it up

Change 297378 merged by Gehel:
spec fix for aptrepo and installserver

https://gerrit.wikimedia.org/r/297378

I have been sprinting that a bit this week. Namely fixed a few spec and rebased Nico patch. Results are under the Gerrit topic rspec-puppet

Change 331854 had a related patch set uploaded (by Hashar):
operations-puppet-rake-jessie: process submodules

https://gerrit.wikimedia.org/r/331854

Change 331854 merged by jenkins-bot:
operations-puppet-rake-jessie: process submodules

https://gerrit.wikimedia.org/r/331854

Change 331856 had a related patch set uploaded (by Hashar):
Jenkins integration of rspec

https://gerrit.wikimedia.org/r/331856

Change 331860 had a related patch set uploaded (by Hashar):
operations-puppet-rake-jessie: read junit results

https://gerrit.wikimedia.org/r/331860

Change 331860 merged by jenkins-bot:
operations-puppet-rake-jessie: read junit results

https://gerrit.wikimedia.org/r/331860

Change 331861 had a related patch set uploaded (by Hashar):
operations-puppet-rake-jessie: fail build on junit error

https://gerrit.wikimedia.org/r/331861

Change 340186 had a related patch set uploaded (by Hashar; owner: Hashar):
Enable rspec testing in Jenkins

https://gerrit.wikimedia.org/r/340186

Change 282484 merged by Filippo Giunchedi:
Rake helper to run rspec in all modules having specs

https://gerrit.wikimedia.org/r/282484

It has taken ages but finally we have some basic setup for running rspec-puppet tests. Next steps will be:

• adding some more documentation, a preliminary draft being https://wikitech.wikimedia.org/wiki/Puppet_coding/testing
• run the spec from Jenkins https://gerrit.wikimedia.org/r/#/c/331856/

Change 340186 abandoned by Hashar:
Enable rspec testing in Jenkins

Reason:
Sorry for the spam of changes. Will squash this one in the parent change https://gerrit.wikimedia.org/r/#/c/331856/

https://gerrit.wikimedia.org/r/340186

In T166888#3360863, @hashar wrote:

In T166888#3333018, @faidon wrote:

<snip>

• Abstract git_changed_in_head into a class, that only runs git commands during initialization. It would then categorize the changed files into buckets (puppet, ruby, erb, python, hiera etc.), and run heuristics for the non-obvious ones (e.g. look at shebangs). The same class could also return all the affected modules, in order to be able to run rspec tests against them.

<snip>
The rspec modules, it is a bit more complicated. They are tightly coupled and the dependencies are listed in .fixtures.yml files at the root of each module. For example the nrpe module has:

modules/nrpe/.fixtures.yml

fixtures:
    symlinks:
        nrpe: "#{source_dir}"
        base: "../../../../base"
        ferm: "../../../../ferm"
        monitoring: "../../../../monitoring"
        wmflib: "../../../../wmflib"

So if a patch touch wmflib, we got to find each modules that define it as a dependency recursively. So that is slightly more complicated.

@Joe proposed a rewriting of the Puppet Rakefile as part of T166888 Patch is https://gerrit.wikimedia.org/r/#/c/366591/ and implements the logic described in previous comment.

Change 331861 abandoned by Hashar:
operations-puppet-rake-jessie: fail build on junit error

https://gerrit.wikimedia.org/r/331861

Change 331856 abandoned by Hashar:
Jenkins integration of rspec

https://gerrit.wikimedia.org/r/331856