Page MenuHomePhabricator

puppet.git rake fails with ruby 2.5
Open, NormalPublic

Description

Tried this today on a Debian Buster system:

$ bundle exec rake test
rake aborted!
NoMethodError: undefined method `<<' for nil:NilClass
/home/godog/src/wikimedia/puppet/vendor/bundle/ruby/2.5.0/gems/puppet-4.8.2/lib/puppet/util/monkey_patches.rb:104:in `<class:SSLContext>'
/home/godog/src/wikimedia/puppet/vendor/bundle/ruby/2.5.0/gems/puppet-4.8.2/lib/puppet/util/monkey_patches.rb:98:in `<top (required)>'
/home/godog/src/wikimedia/puppet/vendor/bundle/ruby/2.5.0/gems/puppet-4.8.2/lib/puppet/util.rb:16:in `require'
/home/godog/src/wikimedia/puppet/vendor/bundle/ruby/2.5.0/gems/puppet-4.8.2/lib/puppet/util.rb:16:in `<module:Util>'
/home/godog/src/wikimedia/puppet/vendor/bundle/ruby/2.5.0/gems/puppet-4.8.2/lib/puppet/util.rb:15:in `<module:Puppet>'
/home/godog/src/wikimedia/puppet/vendor/bundle/ruby/2.5.0/gems/puppet-4.8.2/lib/puppet/util.rb:14:in `<top (required)>'
/home/godog/src/wikimedia/puppet/vendor/bundle/ruby/2.5.0/gems/puppet-4.8.2/lib/puppet.rb:12:in `require'
/home/godog/src/wikimedia/puppet/vendor/bundle/ruby/2.5.0/gems/puppet-4.8.2/lib/puppet.rb:12:in `<top (required)>'
/home/godog/src/wikimedia/puppet/vendor/bundle/ruby/2.5.0/gems/puppet-syntax-2.4.1/lib/puppet-syntax/templates.rb:2:in `require'
/home/godog/src/wikimedia/puppet/vendor/bundle/ruby/2.5.0/gems/puppet-syntax-2.4.1/lib/puppet-syntax/templates.rb:2:in `<top (required)>'
/home/godog/src/wikimedia/puppet/vendor/bundle/ruby/2.5.0/gems/puppet-syntax-2.4.1/lib/puppet-syntax.rb:3:in `require'
/home/godog/src/wikimedia/puppet/vendor/bundle/ruby/2.5.0/gems/puppet-syntax-2.4.1/lib/puppet-syntax.rb:3:in `<top (required)>'
/home/godog/src/wikimedia/puppet/rake_modules/taskgen.rb:9:in `require'
/home/godog/src/wikimedia/puppet/rake_modules/taskgen.rb:9:in `<top (required)>'
/home/godog/src/wikimedia/puppet/Rakefile:38:in `require'
/home/godog/src/wikimedia/puppet/Rakefile:38:in `<top (required)>'
/home/godog/src/wikimedia/puppet/vendor/bundle/ruby/2.5.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
(See full trace by running task with --trace)

The problem seems to be ruby 2.5 and puppet monkey patching ruby ssl, fixed upstream at https://github.com/puppetlabs/puppet/commit/8b112c4bea7f33b98f0753c2d2e09bc231201684 so using puppet gem >= 4.10.2 fixes this

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 2 2018, 8:35 AM

@hashar I initially added this task to ci-infra because it'll be relevant with buster docker/jenkins jobs, is there a task already for that I could piggyback?

hashar added a subscriber: Joe.

The Gemfile uses puppet ~> 4.8.2 which is the version provided by jessie-backports and stretch.

The CI job installs it from rubygems hence we lack monkey patches provided by the Debian package. Recently I had the issue of puppet 4.8.2 lacking the service provider for Stretch (it used runt instead of systemd). @Joe fixed by having our test suite to monkey patch puppet: 077572c50446931fc36943b01015733b12d8c1c5 (and debian package patch.

So I guess our test suite would have to monkey patch and apply https://github.com/puppetlabs/puppet/commit/8b112c4bea7f33b98f0753c2d2e09bc231201684 . I have no idea thought how that can be done.

CDanis triaged this task as Normal priority.Jan 14 2019, 2:41 PM

In case it is useful, on a buster system I'm using this to run rake locally: PUPPET_GEM_VERSION=4.10.12 bundle exec rake test

jbond added a subscriber: jbond.EditedJan 23 2019, 1:55 PM

I have also been trying to get this working the following is also a useful data point https://bugzilla.redhat.com/show_bug.cgi?id=1440710, however applying that fix just causes a different error.

error: Could not send report: undefined method `split' for :ensure:Symbol
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:76:in `block in initialize'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:136:in `accept'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:546:in `block in dump_ivars'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:544:in `each'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:544:in `dump_ivars'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:163:in `visit_Object'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:136:in `accept'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:389:in `block in visit_Enumerator'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:389:in `each'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:389:in `visit_Enumerator'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:381:in `visit_Array'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:136:in `accept'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:535:in `block in emit_coder'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:533:in `each'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:533:in `emit_coder'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:518:in `dump_coder'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:134:in `accept'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:360:in `block in visit_Hash'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:358:in `each'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:358:in `visit_Hash'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:136:in `accept'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:546:in `block in dump_ivars'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:544:in `each'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:544:in `dump_ivars'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:163:in `visit_Object'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:136:in `accept'
/usr/lib/ruby/2.5.0/psych/visitors/yaml_tree.rb:118:in `push'
/usr/lib/ruby/2.5.0/psych.rb:441:in `dump'
/usr/lib/ruby/vendor_ruby/puppet/util/yaml.rb:33:in `block in dump'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:482:in `replace_file'
/usr/lib/ruby/vendor_ruby/puppet/util/yaml.rb:32:in `dump'
/usr/lib/ruby/vendor_ruby/puppet/indirector/yaml.rb:30:in `save'
/usr/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:288:in `save'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:391:in `send_report'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:354:in `run_internal'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:221:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/usr/lib/ruby/vendor_ruby/puppet.rb:241:in `override'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:195:in `run'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:45:in `block (4 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/agent/locker.rb:21:in `lock'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:45:in `block (3 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:98:in `with_client'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:42:in `block (2 levels) in run'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:65:in `run_in_fork'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:41:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:179:in `controlled_run'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:39:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/agent.rb:353:in `onetime'
/usr/lib/ruby/vendor_ruby/puppet/application/agent.rb:331:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:344:in `block in run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:540:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:344:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:132:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute'
/usr/bin/puppet:5:in `<main>'
jbond added a comment.EditedJan 23 2019, 3:34 PM

https://bugzilla.redhat.com/show_bug.cgi?id=1440710, however applying that fix just causes a different error

in fact, the patch mentioned in the commit above is enough to fix the rspec tests. the stack trace i posted is produced when puppet tries to parse its catalogue to be submitted to puppetdb and therefore not relevant to rspec tests

Joe added a comment.EditedJan 24 2019, 9:47 AM

There is a specific reason why the recommended method for running our spec tests includes the use of rbenv https://wikitech.wikimedia.org/wiki/Puppet_coding#Install_rbenv_and_python_tox_packages :

tests should be run in the same environment as production, so with the same ruby version[1], the same puppet version, etc. When we'll move to ruby 2.5, we'll probably also move to puppet 5+, which I guess is going to be a much larger problem.

I don't think there is anything we should fix here.
[1] This is almost true, the ruby version shipped with debian package gets quite a few security fixes backported, but this is the next best alternative to running tests in a container.

I 've uploaded to 4.10.2 today using https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/524525/ fwiw. I had not seen this task, sorry about that. Feel free to revert if I caused any problems (altough a fleet wide PCC said ok)

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptJul 25 2019, 3:39 PM
jbond added a comment.Jul 25 2019, 4:33 PM

In case this needs rolling back the the issue can be fixed in 4.8 with the following patch https://phabricator.wikimedia.org/P8772

fgiunchedi closed this task as Resolved.Jul 26 2019, 7:48 AM
fgiunchedi claimed this task.

The original issue is gone (local tests on Buster), being bold and resolving the task for now but feel free to reopen, thanks everyone!

hashar reopened this task as Open.Jul 28 2019, 8:17 PM

The Gemfile had Puppet 4.8.2 to match the version provided by Debian Jessie:

Gemfile
gem 'puppet', ENV['PUPPET_GEM_VERSION'] || '~> 4.8.2'

Though it lacks some patches made by Debian, notably one that to make services use systemd (our patch is rake_modules/fix_service_provider.rb).

Apparently we now use Puppet 5.5.10 across the fleet, so we can probably update the Gemfile to match the version?

The Gemfile had Puppet 4.8.2 to match the version provided by Debian Jessie:

Stretch, not jessie. jessie shipped with 3.7.2

Gemfile
gem 'puppet', ENV['PUPPET_GEM_VERSION'] || '~> 4.8.2'

Though it lacks some patches made by Debian, notably one that to make services use systemd (our patch is rake_modules/fix_service_provider.rb).

Yup, that monkey patch still holds though, doesn't it? I mean it might not be needed anymore (haven't checked), but it sure still is applied.

Apparently we now use Puppet 5.5.10 across the fleet, so we can probably update the Gemfile to match the version?

5.5.10 is the puppet agent, the puppetmasters still run 4.8.2. There is a goal (see T228657) to do that upgrade this quarter, but we still are on 4.8.2.

I guess though it wouldn't hurt to bump CI to 5.5.10. That should potentially save us from some issues in production.

Change 526104 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Bump CI puppet Gem version to 5.5.10

https://gerrit.wikimedia.org/r/526104

The Gemfile had Puppet 4.8.2 to match the version provided by Debian Jessie:

Stretch, not jessie. jessie shipped with 3.7.2.

Indeed sorry. And I can not remember how we have handled the transition from Puppet 3.7 to Puppet 4.8 CI wise :-(

Gemfile
gem 'puppet', ENV['PUPPET_GEM_VERSION'] || '~> 4.8.2'

Though it lacks some patches made by Debian, notably one that to make services use systemd (our patch is rake_modules/fix_service_provider.rb).

Yup, that monkey patch still holds though, doesn't it? I mean it might not be needed anymore (haven't checked), but it sure still is applied.

I can't tell, it is a bit messy in puppet code :]

puppet-5.5.16/lib/puppet/provider/service/systemd.rb
27   defaultfor :operatingsystem => :debian, :operatingsystemmajrelease => ["8", "stretch/sid", "9", "buster/sid"]

Anyway that was just a side issue only happening when running rspec-puppet on a Stretch machine. I don't think that caused any troubles in production.

Apparently we now use Puppet 5.5.10 across the fleet, so we can probably update the Gemfile to match the version?

5.5.10 is the puppet agent, the puppetmasters still run 4.8.2. There is a goal (see T228657) to do that upgrade this quarter, but we still are on 4.8.2.
I guess though it wouldn't hurt to bump CI to 5.5.10. That should potentially save us from some issues in production.

I guess we we will want the CI container to use the same distro and puppet version than our puppet master. And even use the Debian package. I imagine lint / spec might fail at some point during the migration, and I guess we would want to run tests with both versions.

Maybe it is a better to fill all of that as sub tasks of the migration task T228657 :]