Presto cluster online and usable with test data pushed from analytics prod infrastructure accessible by Cloud (labs) users
Open, HighPublic

Description

When this task is completed we will have a 3 node presto cluster in labs with tests data. This data is been pushed from one of the mediawiki reconstruction snapshots we hold in hadoop, the test data is available to be queried by users of our cloud environment. The task is completed when infrastructure is setup and useable.

Nuria created this task.Sep 20 2018, 1:52 PM
Nuria renamed this task from Presto cluster online and useable with test data pushed from analytics prod infrastructure accessible by Cloud (labs) users to Presto cluster online and usable with test data pushed from analytics prod infrastructure accessible by Cloud (labs) users.Sep 20 2018, 4:06 PM
Milimetric triaged this task as High priority.

Change 467766 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet/cdh@master] Use net_topology script content rather than erb path

https://gerrit.wikimedia.org/r/467766

Change 467769 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Move Hadoop net topology to hiera

https://gerrit.wikimedia.org/r/467769

Change 467815 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Move Hive profile settings to role hiera

https://gerrit.wikimedia.org/r/467815

Change 467821 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Label Hadoop prometheus metrics with the hadoop_cluster_name

https://gerrit.wikimedia.org/r/467821

Change 467766 merged by Ottomata:
[operations/puppet/cdh@master] Use net_topology script content rather than erb path

https://gerrit.wikimedia.org/r/467766

Change 467815 merged by Ottomata:
[operations/puppet@production] Move Hive profile settings to role hiera

https://gerrit.wikimedia.org/r/467815

Change 467769 merged by Ottomata:
[operations/puppet@production] Move Hadoop net topology to hiera

https://gerrit.wikimedia.org/r/467769

Change 468004 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Temporarily revert the change to net-topology.py.erb

https://gerrit.wikimedia.org/r/468004

Change 468004 merged by Ottomata:
[operations/puppet@production] Temporarily revert the change to net-topology.py.erb

https://gerrit.wikimedia.org/r/468004

Change 468046 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet/cdh@master] Use net_topology_script_path to configure net.topology.script.file.name

https://gerrit.wikimedia.org/r/468046

Change 468046 merged by Ottomata:
[operations/puppet/cdh@master] Use net_topology_script_path to configure net.topology.script.file.name

https://gerrit.wikimedia.org/r/468046

Change 468049 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix net-topology.py.erb script to render proper python dict

https://gerrit.wikimedia.org/r/468049

Change 468049 merged by Ottomata:
[operations/puppet@production] Fix net-topology.py.erb script to render proper python dict

https://gerrit.wikimedia.org/r/468049

Change 467821 merged by Ottomata:
[operations/puppet@production] Label Hadoop prometheus metrics with the hadoop_cluster_name

https://gerrit.wikimedia.org/r/467821

Change 468056 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add more labels to Hadoop daemon JMX prometheus metrics

https://gerrit.wikimedia.org/r/468056

Change 468068 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use hadoop_cluster label in icinga alerts

https://gerrit.wikimedia.org/r/468068

Change 468068 merged by Ottomata:
[operations/puppet@production] Use hadoop_cluster label in icinga alerts

https://gerrit.wikimedia.org/r/468068

Change 468070 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] Configure cloud-analytics-eqiad Hadoop cluster

https://gerrit.wikimedia.org/r/468070

Change 468071 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Conditionally pass in zookeeper_hosts to cdh::hive

https://gerrit.wikimedia.org/r/468071

Change 468071 merged by Ottomata:
[operations/puppet@production] Conditionally pass in zookeeper_hosts to cdh::hive

https://gerrit.wikimedia.org/r/468071

Milimetric moved this task from Next Up to In Progress on the Analytics-Kanban board.

Change 468056 abandoned by Ottomata:
Add more labels to Hadoop daemon JMX prometheus metrics

https://gerrit.wikimedia.org/r/468056

Change 478746 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add hieradata/labs/cloud-analytics/common.yaml

https://gerrit.wikimedia.org/r/478746

Change 478746 merged by Ottomata:
[operations/puppet@production] Add hieradata/labs/cloud-analytics/common.yaml

https://gerrit.wikimedia.org/r/478746

Change 478748 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add cloud-analytics zookeeper settings

https://gerrit.wikimedia.org/r/478748

Change 478748 merged by Ottomata:
[operations/puppet@production] Add cloud-analytics zookeeper settings

https://gerrit.wikimedia.org/r/478748

Change 478751 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Update zookeeper package version for cloud-analytics

https://gerrit.wikimedia.org/r/478751

Change 478751 merged by Ottomata:
[operations/puppet@production] Update zookeeper package version for cloud-analytics

https://gerrit.wikimedia.org/r/478751

Change 478755 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Allow configuration of $hadoop_var_directory in profile::hadoop::commmon

https://gerrit.wikimedia.org/r/478755

Change 478755 merged by Ottomata:
[operations/puppet@production] Allow configuration of $hadoop_var_directory in profile::hadoop::commmon

https://gerrit.wikimedia.org/r/478755

Change 478758 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use subdir of hadoop data path for datanode_mounts in cloud-analytics

https://gerrit.wikimedia.org/r/478758

Change 478758 merged by Ottomata:
[operations/puppet@production] Use subdir of hadoop data path for datanode_mounts in cloud-analytics

https://gerrit.wikimedia.org/r/478758

Change 478767 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Set monitoring_enabled to true for cloud-analytics

https://gerrit.wikimedia.org/r/478767

Change 478767 merged by Ottomata:
[operations/puppet@production] Set monitoring_enabled to true for cloud-analytics

https://gerrit.wikimedia.org/r/478767

Change 478772 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Bump zookeeper submodule version

https://gerrit.wikimedia.org/r/478772

Change 478772 merged by Ottomata:
[operations/puppet@production] Bump zookeeper submodule version

https://gerrit.wikimedia.org/r/478772

Change 478778 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Attempt to get around prometheus jmx exporter race condition on new cluster

https://gerrit.wikimedia.org/r/478778

Change 478778 merged by Ottomata:
[operations/puppet@production] Attempt to get around prometheus jmx exporter race condition on new cluster

https://gerrit.wikimedia.org/r/478778

Change 478779 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Undo last change and temporarily disable JMX prometheus in cloud-analytics

https://gerrit.wikimedia.org/r/478779

Change 478779 merged by Ottomata:
[operations/puppet@production] Undo last change and temporarily disable JMX prometheus in cloud-analytics

https://gerrit.wikimedia.org/r/478779

Change 478782 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Bump cdh submodule to vary zkCli.sh path

https://gerrit.wikimedia.org/r/478782

Change 478782 merged by Ottomata:
[operations/puppet@production] Bump cdh submodule to vary zkCli.sh path

https://gerrit.wikimedia.org/r/478782

Change 478789 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Move cloud-analytics zookeeper to ca-conf-*

https://gerrit.wikimedia.org/r/478789

Change 478789 merged by Ottomata:
[operations/puppet@production] Move cloud-analytics zookeeper to ca-conf-*

https://gerrit.wikimedia.org/r/478789

Change 478798 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Set yarn and hadoop heapsize for cloud-analytics

https://gerrit.wikimedia.org/r/478798

Change 478798 merged by Ottomata:
[operations/puppet@production] Set yarn and hadoop heapsize for cloud-analytics

https://gerrit.wikimedia.org/r/478798

Change 478800 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Set Xmx for datanode and nodemanager in cloud-analytics

https://gerrit.wikimedia.org/r/478800

Change 478800 merged by Ottomata:
[operations/puppet@production] Set Xmx for datanode and nodemanager in cloud-analytics

https://gerrit.wikimedia.org/r/478800

Change 478801 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Reenable hadoop prometheus jmx exporters in cloud-analytics

https://gerrit.wikimedia.org/r/478801

Change 478801 merged by Ottomata:
[operations/puppet@production] Reenable hadoop prometheus jmx exporters in cloud-analytics

https://gerrit.wikimedia.org/r/478801

Status update!

cloud-analytics Hadoop cluster is up and running! Tomorrow will be Presto work... :)

\o/ !!! That's super great :)
Let's discuss a one-off way to copy data over for tests.
Is there another another way than through public-endpot?
If not, we shold start the copy :)

Change 478962 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix comment in cloud-analytics common.yaml

https://gerrit.wikimedia.org/r/478962

Change 478963 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add cloud_analytics roles

https://gerrit.wikimedia.org/r/478963

Change 478962 merged by Ottomata:
[operations/puppet@production] Fix comment in cloud-analytics common.yaml

https://gerrit.wikimedia.org/r/478962

Change 478963 merged by Ottomata:
[operations/puppet@production] Add cloud_analytics roles

https://gerrit.wikimedia.org/r/478963

Change 468070 abandoned by Ottomata:
[WIP] Configure cloud-analytics-eqiad Hadoop cluster

Reason:
doing this in Cloud VPS

https://gerrit.wikimedia.org/r/468070

Change 479324 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use fqdn instead of hostname in nodes id

https://gerrit.wikimedia.org/r/479324

Change 479324 merged by Ottomata:
[operations/puppet@production] Use fqdn instead of hostname in nodes id

https://gerrit.wikimedia.org/r/479324

Change 479339 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Set presto hive connector name to hive-hadoop2

https://gerrit.wikimedia.org/r/479339

Change 479339 merged by Ottomata:
[operations/puppet@production] Set presto hive connector name to hive-hadoop2

https://gerrit.wikimedia.org/r/479339

Change 479343 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use thrift:// uri for hive http:// uri for discovery uri

https://gerrit.wikimedia.org/r/479343

Change 479343 merged by Ottomata:
[operations/puppet@production] Use thrift:// uri for hive http:// uri for discovery uri

https://gerrit.wikimedia.org/r/479343