Page MenuHomePhabricator

Run Atlas on cloud services cluster
Open, Needs TriagePublic

Description

The node for data catalog evaluations is: data-catalog-evaluation.analytics.eqiad1.wikimedia.cloud

Currently Atlas can be accessed via

ssh -NL 21000:data-catalog-evaluation.analytics.eqiad1.wikimedia.cloud:21000 data-catalog-evaluation.analytics.eqiad1.wikimedia.cloud

but it would be nice to have a persistent hostname like atlas-evaluation.wmfcloud.org as well.

Event Timeline

Currently adding as a hive docker container

Found a dockerfile for hive: https://github.com/IBM/docker-hive/

Built a docker image:

razzi@data-catalog-evaluation:~/mnt/docker-hive$ docker build . -t hive-metastore:upstream

Ran the docker image:

docker run -d -p 9093:9093 hive-metastore:upstream (produced container id 8d66326230ead4)

Connected to docker image:

razzi@data-catalog-evaluation:~/mnt/docker-hive$ docker exec -it 8d66326230ead4 /bin/bash

Checked if port was open with curl:

hive@8d66326230ea:~$ curl localhost:9083
curl: (52) Empty reply from server

It gives an error, but it connects (good). In the hive logs, I see an error which indicates it got the request (but it doesn't speak http)

2022-01-13T22:46:54,844 ERROR [pool-8-thread-6] server.TThreadPoolServer: Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in readMessageBegin, old client?

However from the host machine curl did not connect:

razzi@data-catalog-evaluation:~$ curl localhost:9083
curl: (7) Failed to connect to localhost port 9083: Connection refused

This is strange because I have port forwarding from 9083. Since the host machine can't even connect to hive, atlas won't be able to since it's running in a separate docker container.

Incorporating the hive docker image into the atlas docker-compose.yaml (data-catalog-evaluation.analytics.eqiad1.wikimedia.cloud:/home/razzi/apache-atlas-docker/docker-compose.yml) would probably work.