Page MenuHomePhabricator

role::spark should compile into a catalogue without dependency cycles
Open, LowPublic

Description

2) role::spark should compile into a catalogue without dependency cycles
   Failure/Error: it { is_expected.to compile }
     error during compilation: Could not find resource 'Class[Cdh::Hive]' for relationship on 'Class[Cdh::Spark]' on node wmf-bd808-mbp01.local
   # ./puppet/spec/integration/roles_spec.rb:20:in `block (3 levels) in <top (required)>'

Event Timeline

bd808 created this task.Jan 4 2018, 4:57 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 4 2018, 4:57 AM
bd808 added a subscriber: Ottomata.

Poking a bit deeper, there are a whole pile of things that need to be fixed with these roles:

  • analytics
  • hadoop
  • hive
  • hue
  • oozie
  • spark

The version of puppet/modules/cdh that is used in MediaWiki-Vagrant is quite a bit behind the version used in operations/puppet.git. Some resources have been renamed (e.g. ::cdh::hive::master -> ::cdh::hive::server). There are also package dependencies that conflict with core packages installed in MediaWiki-Vagrant:

==> default: Error: Evaluation Error: Error while evaluating a Resource Statement, Duplicate declaration: Package[openssl] is already declared in file /vagrant/puppet/modules/cdh/manifests/hue.pp:189; cannot redeclare at /vagrant/puppet/modules/postfix/manifests/init.pp:11 at /vagrant/puppet/modules/postfix/manifests/init.pp:11:5 on node vagrantdisposable.mediawiki-vagrant.dev

It looks like it may have been a couple years since @Ottomata or really anyone else touched these roles.

bd808 triaged this task as Low priority.Jan 7 2018, 7:38 PM

I think we should remove all hadoop/cdh related puppet from mediawiki-vagrant :)

fdans moved this task from Incoming to Radar on the Analytics board.Jan 8 2018, 4:43 PM

Change 403341 had a related patch set uploaded (by BryanDavis; owner: Bryan Davis):
[mediawiki/vagrant@stretch-migration] analytics: remove cdh modules and roles

https://gerrit.wikimedia.org/r/403341

Change 403341 merged by jenkins-bot:
[mediawiki/vagrant@stretch-migration] analytics: remove cdh modules and roles

https://gerrit.wikimedia.org/r/403341