@hashar I initially added this task to ci-infra because it'll be relevant with buster docker/jenkins jobs, is there a task already for that I could piggyback?

The Gemfile uses puppet ~> 4.8.2 which is the version provided by jessie-backports and stretch.

The CI job installs it from rubygems hence we lack monkey patches provided by the Debian package. Recently I had the issue of puppet 4.8.2 lacking the service provider for Stretch (it used runt instead of systemd). @Joe fixed by having our test suite to monkey patch puppet: 077572c50446931fc36943b01015733b12d8c1c5 (and debian package patch.

So I guess our test suite would have to monkey patch and apply https://github.com/puppetlabs/puppet/commit/8b112c4bea7f33b98f0753c2d2e09bc231201684 . I have no idea thought how that can be done.

In case it is useful, on a buster system I'm using this to run rake locally: PUPPET_GEM_VERSION=4.10.12 bundle exec rake test

I have also been trying to get this working the following is also a useful data point https://bugzilla.redhat.com/show_bug.cgi?id=1440710, however applying that fix just causes a different error.

error: Could not send report: undefined method `split' for :ensure:Symbol
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:76:in `block in initialize'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:136:in `accept'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:546:in `block in dump_ivars'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:544:in `each'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:544:in `dump_ivars'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:163:in `visit_Object'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:136:in `accept'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:389:in `block in visit_Enumerator'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:389:in `each'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:389:in `visit_Enumerator'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:381:in `visit_Array'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:136:in `accept'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:535:in `block in emit_coder'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:533:in `each'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:533:in `emit_coder'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:518:in `dump_coder'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:134:in `accept'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:360:in `block in visit_Hash'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:358:in `each'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:358:in `visit_Hash'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:136:in `accept'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:546:in `block in dump_ivars'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:544:in `each'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:544:in `dump_ivars'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:163:in `visit_Object'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:136:in `accept'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:118:in `push'
/usr/lib/ruby/2.5.0/psych.rb:441:in `dump'
/usr/lib/ruby/vendor_ruby/puppet/util/yaml.rb:33:in `block in dump'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:482:in `replace_file'
/usr/lib/ruby/vendor_ruby/puppet/util/yaml.rb:32:in `dump'
/usr/lib/ruby/vendor_ruby/puppet/indirector/yaml.rb:30:in `save'
/usr/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:288:in `save'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:391:in `send_report'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:354:in `run_internal'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:221:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:241:in `override'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:195:in `run'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:45:in `block (4 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/agent/locker.rb:21:in `lock'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:45:in `block (3 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:98:in `with_client'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:42:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:65:in `run_in_fork'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:41:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:179:in `controlled_run'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:39:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/agent.rb:353:in `onetime'
/usr/lib/ruby/vendor_ruby/puppet/application/agent.rb:331:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:344:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:540:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:344:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:132:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute'
/usr/bin/puppet:5:in `<main>'

https://bugzilla.redhat.com/show_bug.cgi?id=1440710, however applying that fix just causes a different error

in fact, the patch mentioned in the commit above is enough to fix the rspec tests. the stack trace i posted is produced when puppet tries to parse its catalogue to be submitted to puppetdb and therefore not relevant to rspec tests

There is a specific reason why the recommended method for running our spec tests includes the use of rbenv https://wikitech.wikimedia.org/wiki/Puppet_coding#Install_rbenv_and_python_tox_packages :

tests should be run in the same environment as production, so with the same ruby version[1], the same puppet version, etc. When we'll move to ruby 2.5, we'll probably also move to puppet 5+, which I guess is going to be a much larger problem.

I don't think there is anything we should fix here.
[1] This is almost true, the ruby version shipped with debian package gets quite a few security fixes backported, but this is the next best alternative to running tests in a container.

I 've uploaded to 4.10.2 today using https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/524525/ fwiw. I had not seen this task, sorry about that. Feel free to revert if I caused any problems (altough a fleet wide PCC said ok)

In case this needs rolling back the the issue can be fixed in 4.8 with the following patch https://phabricator.wikimedia.org/P8772

In T208566#5371633, @hashar wrote:

The Gemfile had Puppet 4.8.2 to match the version provided by Debian Jessie:

Stretch, not jessie. jessie shipped with 3.7.2

Gemfile

gem 'puppet', ENV['PUPPET_GEM_VERSION'] || '~> 4.8.2'

Though it lacks some patches made by Debian, notably one that to make services use systemd (our patch is rake_modules/fix_service_provider.rb).

Yup, that monkey patch still holds though, doesn't it? I mean it might not be needed anymore (haven't checked), but it sure still is applied.

Apparently we now use Puppet 5.5.10 across the fleet, so we can probably update the Gemfile to match the version?

5.5.10 is the puppet agent, the puppetmasters still run 4.8.2. There is a goal (see T228657) to do that upgrade this quarter, but we still are on 4.8.2.

I guess though it wouldn't hurt to bump CI to 5.5.10. That should potentially save us from some issues in production.

Change 526104 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Bump CI puppet Gem version to 5.5.10

https://gerrit.wikimedia.org/r/526104

In T208566#5372005, @akosiaris wrote:

In T208566#5371633, @hashar wrote:

The Gemfile had Puppet 4.8.2 to match the version provided by Debian Jessie:

Stretch, not jessie. jessie shipped with 3.7.2.

Indeed sorry. And I can not remember how we have handled the transition from Puppet 3.7 to Puppet 4.8 CI wise :-(

Gemfile

gem 'puppet', ENV['PUPPET_GEM_VERSION'] || '~> 4.8.2'

Though it lacks some patches made by Debian, notably one that to make services use systemd (our patch is rake_modules/fix_service_provider.rb).

Yup, that monkey patch still holds though, doesn't it? I mean it might not be needed anymore (haven't checked), but it sure still is applied.

I can't tell, it is a bit messy in puppet code :]

27   defaultfor :operatingsystem => :debian, :operatingsystemmajrelease => ["8", "stretch/sid", "9", "buster/sid"]

Anyway that was just a side issue only happening when running rspec-puppet on a Stretch machine. I don't think that caused any troubles in production.

Apparently we now use Puppet 5.5.10 across the fleet, so we can probably update the Gemfile to match the version?

5.5.10 is the puppet agent, the puppetmasters still run 4.8.2. There is a goal (see T228657) to do that upgrade this quarter, but we still are on 4.8.2.

I guess though it wouldn't hurt to bump CI to 5.5.10. That should potentially save us from some issues in production.

I guess we we will want the CI container to use the same distro and puppet version than our puppet master. And even use the Debian package. I imagine lint / spec might fail at some point during the migration, and I guess we would want to run tests with both versions.

Maybe it is a better to fill all of that as sub tasks of the migration task T228657 :]

Change 545289 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] CI rspec: update puppet version used in spec tests

https://gerrit.wikimedia.org/r/545289

Change 545289 merged by Jbond:
[operations/puppet@production] CI rspec: update puppet version used in spec tests

https://gerrit.wikimedia.org/r/545289

Change 526104 abandoned by Hashar:
Bump CI puppet Gem version to 5.5.10

Reason:
Been done by John Bond eventually 7af791b669fbd815b563cca263b646b8784f0065

https://gerrit.wikimedia.org/r/526104