Page MenuHomePhabricator

Make it (just a bit) easier to spin up Hadoop cluster in Cloud VMs
Closed, DeclinedPublic

Description

I just set up a simple Hadoop-only cluster in deployment-prep using profile::hadoop::master and profile::hadoop::worker. This required a few hiera settings that I don't think should be necessary for simple non-HA Hadoop.

  • zookeeper_clusters, hadoop_clusters.<name>.zookeeper_cluster_name - I don't think these are always required for simple Hadoop. We shouldn't need to use (and set up) Zookeeper if we don't have to.
  • profile::hive::client::{server_host,server_port,hive_metastore_host - profile::hadoop::worker always includes profile::hive::client. We should avoid including this if e.g. profile::hive::client::server_host isn't set.