Page MenuHomePhabricator

TLS connection for hive-standalone-metaserver with minio
Closed, ResolvedPublic

Description

The goal was to get hive-standalone-metaserver working with minio for TLS connections. This was straightforward with trino but hive was a challenge. This task documents what we learned for posterity. Information below could be incorrect, but it's the best I've been able to scrape together from available documentation plus many bug tasks and forum posts from others trying to do the same.

minio supports TLSv1.2 and TLSv1.3, and as of today a fairly small set of ciphers by default. They are promoting security features like forward secrecy. This can be adjusted by setting "MINIO_API_SECURE_CIPHERS=off" in the startup environment. In our setup, this is in /etc/default/minio which is sourced by the minio systemd unit file. As of today, hive-standalone-metastore appears to be stuck on java 8, TLSv1.2, and an archaic collection of ciphers. There is only one cipher that overlaps with the list supported by minio by default, and minio only supports that cipher on TLSv1.3. So the two services are unable to negotiate a connection. We figured this out by enabling debug logging for hive-standalone-metastore, (HADOOP_OPTS="-Djavax.net.debug=all" in the startup environment). minio doesn't really 'do' logging, not much to go on from that end of the connection.

So the first thing we had to fix was setting MINIO_API_SECURE_CIPHERS=off for minio side to extend the TLSv1.2 cipher list.

The second problem was getting hive-standalone-metastore to use our internal CA certificate in its truststore. There were a lot of false starts on this, we haven't found a setting that overrides temurin's truststore location though this appears to be possible with other jdks. Until there's a better approach we replaced the package installed cacerts file. Note that the 'changeit' password is a java convention and this truststore only contains one CA cert.

$in_ca = '/var/lib/puppet/ssl/certs/ca.pem'

exec { 'ssl::puppet_adoptium_truststore':
    command => "/usr/bin/keytool -importcert -cacerts -storepass changeit -file ${in_ca} -alias puppet -trustcacerts -noprompt",
    unless  => '/usr/bin/keytool -list -cacerts -storepass changeit|grep ^puppet',
}

Event Timeline

Jgreen updated the task description. (Show Details)
Jgreen moved this task from Backlog to Completed on the Fundraising analytics stack board.
Jgreen added a project: fundraising-tech-ops.

This is deployed.