Page MenuHomePhabricator

Presto cluster online and usable with test data pushed from analytics prod infrastructure accessible by Cloud (labs) users
Open, HighPublic13 Story Points

Description

When this task is completed we will have a 3 node presto cluster in labs with tests data. This data is been pushed from one of the mediawiki reconstruction snapshots we hold in hadoop, the test data is available to be queried by users of our cloud environment. The task is completed when infrastructure is setup and useable.

Details

Related Gerrit Patches:
operations/puppet : productionFix Hadoop settings for presto in cloud-analytics
operations/puppet : productionMake presto module and profile smarter
operations/puppet : productionAdd ::profile::presto::server to ca-master-2 to run Presto coordinator
operations/puppet : productionRun Presto coordinator on ca-master-2 in cloud-analytics
operations/puppet : productionUse thrift:// uri for hive http:// uri for discovery uri
operations/puppet : productionSet presto hive connector name to hive-hadoop2
operations/puppet : productionUse fqdn instead of hostname in nodes id
operations/puppet : production[WIP] Configure cloud-analytics-eqiad Hadoop cluster
operations/puppet : productionAdd cloud_analytics roles
operations/puppet : productionFix comment in cloud-analytics common.yaml
operations/puppet : productionReenable hadoop prometheus jmx exporters in cloud-analytics
operations/puppet : productionSet Xmx for datanode and nodemanager in cloud-analytics
operations/puppet : productionSet yarn and hadoop heapsize for cloud-analytics
operations/puppet : productionMove cloud-analytics zookeeper to ca-conf-*
operations/puppet : productionBump cdh submodule to vary zkCli.sh path
operations/puppet : productionUndo last change and temporarily disable JMX prometheus in cloud-analytics
operations/puppet : productionAttempt to get around prometheus jmx exporter race condition on new cluster
operations/puppet : productionBump zookeeper submodule version
operations/puppet : productionSet monitoring_enabled to true for cloud-analytics
operations/puppet : productionUse subdir of hadoop data path for datanode_mounts in cloud-analytics
operations/puppet : productionAllow configuration of $hadoop_var_directory in profile::hadoop::commmon
operations/puppet : productionUpdate zookeeper package version for cloud-analytics
operations/puppet : productionAdd cloud-analytics zookeeper settings
operations/puppet : productionAdd hieradata/labs/cloud-analytics/common.yaml
operations/puppet : productionAdd more labels to Hadoop daemon JMX prometheus metrics
operations/puppet : productionConditionally pass in zookeeper_hosts to cdh::hive
operations/puppet : productionUse hadoop_cluster label in icinga alerts
operations/puppet : productionLabel Hadoop prometheus metrics with the hadoop_cluster_name
operations/puppet : productionFix net-topology.py.erb script to render proper python dict
operations/puppet/cdh : masterUse net_topology_script_path to configure net.topology.script.file.name
operations/puppet : productionTemporarily revert the change to net-topology.py.erb
operations/puppet : productionMove Hadoop net topology to hiera
operations/puppet : productionMove Hive profile settings to role hiera
operations/puppet/cdh : masterUse net_topology script content rather than erb path

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 467766 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet/cdh@master] Use net_topology script content rather than erb path

https://gerrit.wikimedia.org/r/467766

Change 467769 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Move Hadoop net topology to hiera

https://gerrit.wikimedia.org/r/467769

Change 467815 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Move Hive profile settings to role hiera

https://gerrit.wikimedia.org/r/467815

Change 467821 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Label Hadoop prometheus metrics with the hadoop_cluster_name

https://gerrit.wikimedia.org/r/467821

Change 467766 merged by Ottomata:
[operations/puppet/cdh@master] Use net_topology script content rather than erb path

https://gerrit.wikimedia.org/r/467766

Change 467815 merged by Ottomata:
[operations/puppet@production] Move Hive profile settings to role hiera

https://gerrit.wikimedia.org/r/467815

Change 467769 merged by Ottomata:
[operations/puppet@production] Move Hadoop net topology to hiera

https://gerrit.wikimedia.org/r/467769

Change 468004 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Temporarily revert the change to net-topology.py.erb

https://gerrit.wikimedia.org/r/468004

Change 468004 merged by Ottomata:
[operations/puppet@production] Temporarily revert the change to net-topology.py.erb

https://gerrit.wikimedia.org/r/468004

Change 468046 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet/cdh@master] Use net_topology_script_path to configure net.topology.script.file.name

https://gerrit.wikimedia.org/r/468046

Change 468046 merged by Ottomata:
[operations/puppet/cdh@master] Use net_topology_script_path to configure net.topology.script.file.name

https://gerrit.wikimedia.org/r/468046

Change 468049 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix net-topology.py.erb script to render proper python dict

https://gerrit.wikimedia.org/r/468049

Change 468049 merged by Ottomata:
[operations/puppet@production] Fix net-topology.py.erb script to render proper python dict

https://gerrit.wikimedia.org/r/468049

Change 467821 merged by Ottomata:
[operations/puppet@production] Label Hadoop prometheus metrics with the hadoop_cluster_name

https://gerrit.wikimedia.org/r/467821

Change 468056 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add more labels to Hadoop daemon JMX prometheus metrics

https://gerrit.wikimedia.org/r/468056

Change 468068 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use hadoop_cluster label in icinga alerts

https://gerrit.wikimedia.org/r/468068

Change 468068 merged by Ottomata:
[operations/puppet@production] Use hadoop_cluster label in icinga alerts

https://gerrit.wikimedia.org/r/468068

Change 468070 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] Configure cloud-analytics-eqiad Hadoop cluster

https://gerrit.wikimedia.org/r/468070

Change 468071 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Conditionally pass in zookeeper_hosts to cdh::hive

https://gerrit.wikimedia.org/r/468071

Change 468071 merged by Ottomata:
[operations/puppet@production] Conditionally pass in zookeeper_hosts to cdh::hive

https://gerrit.wikimedia.org/r/468071

Milimetric moved this task from Next Up to In Progress on the Analytics-Kanban board.

Change 468056 abandoned by Ottomata:
Add more labels to Hadoop daemon JMX prometheus metrics

https://gerrit.wikimedia.org/r/468056

Change 478746 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add hieradata/labs/cloud-analytics/common.yaml

https://gerrit.wikimedia.org/r/478746

Change 478746 merged by Ottomata:
[operations/puppet@production] Add hieradata/labs/cloud-analytics/common.yaml

https://gerrit.wikimedia.org/r/478746

Change 478748 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add cloud-analytics zookeeper settings

https://gerrit.wikimedia.org/r/478748

Change 478748 merged by Ottomata:
[operations/puppet@production] Add cloud-analytics zookeeper settings

https://gerrit.wikimedia.org/r/478748

Change 478751 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Update zookeeper package version for cloud-analytics

https://gerrit.wikimedia.org/r/478751

Change 478751 merged by Ottomata:
[operations/puppet@production] Update zookeeper package version for cloud-analytics

https://gerrit.wikimedia.org/r/478751

Change 478755 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Allow configuration of $hadoop_var_directory in profile::hadoop::commmon

https://gerrit.wikimedia.org/r/478755

Change 478755 merged by Ottomata:
[operations/puppet@production] Allow configuration of $hadoop_var_directory in profile::hadoop::commmon

https://gerrit.wikimedia.org/r/478755

Change 478758 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use subdir of hadoop data path for datanode_mounts in cloud-analytics

https://gerrit.wikimedia.org/r/478758

Change 478758 merged by Ottomata:
[operations/puppet@production] Use subdir of hadoop data path for datanode_mounts in cloud-analytics

https://gerrit.wikimedia.org/r/478758

Change 478767 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Set monitoring_enabled to true for cloud-analytics

https://gerrit.wikimedia.org/r/478767

Change 478767 merged by Ottomata:
[operations/puppet@production] Set monitoring_enabled to true for cloud-analytics

https://gerrit.wikimedia.org/r/478767

Change 478772 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Bump zookeeper submodule version

https://gerrit.wikimedia.org/r/478772

Change 478772 merged by Ottomata:
[operations/puppet@production] Bump zookeeper submodule version

https://gerrit.wikimedia.org/r/478772

Change 478778 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Attempt to get around prometheus jmx exporter race condition on new cluster

https://gerrit.wikimedia.org/r/478778

Change 478778 merged by Ottomata:
[operations/puppet@production] Attempt to get around prometheus jmx exporter race condition on new cluster

https://gerrit.wikimedia.org/r/478778

Change 478779 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Undo last change and temporarily disable JMX prometheus in cloud-analytics

https://gerrit.wikimedia.org/r/478779

Change 478779 merged by Ottomata:
[operations/puppet@production] Undo last change and temporarily disable JMX prometheus in cloud-analytics

https://gerrit.wikimedia.org/r/478779

Change 478782 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Bump cdh submodule to vary zkCli.sh path

https://gerrit.wikimedia.org/r/478782

Change 478782 merged by Ottomata:
[operations/puppet@production] Bump cdh submodule to vary zkCli.sh path

https://gerrit.wikimedia.org/r/478782

Change 478789 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Move cloud-analytics zookeeper to ca-conf-*

https://gerrit.wikimedia.org/r/478789

Change 478789 merged by Ottomata:
[operations/puppet@production] Move cloud-analytics zookeeper to ca-conf-*

https://gerrit.wikimedia.org/r/478789

Change 478798 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Set yarn and hadoop heapsize for cloud-analytics

https://gerrit.wikimedia.org/r/478798

Change 478798 merged by Ottomata:
[operations/puppet@production] Set yarn and hadoop heapsize for cloud-analytics

https://gerrit.wikimedia.org/r/478798

Change 478800 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Set Xmx for datanode and nodemanager in cloud-analytics

https://gerrit.wikimedia.org/r/478800

Change 478800 merged by Ottomata:
[operations/puppet@production] Set Xmx for datanode and nodemanager in cloud-analytics

https://gerrit.wikimedia.org/r/478800

Change 478801 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Reenable hadoop prometheus jmx exporters in cloud-analytics

https://gerrit.wikimedia.org/r/478801

Change 478801 merged by Ottomata:
[operations/puppet@production] Reenable hadoop prometheus jmx exporters in cloud-analytics

https://gerrit.wikimedia.org/r/478801

Status update!

cloud-analytics Hadoop cluster is up and running! Tomorrow will be Presto work... :)

\o/ !!! That's super great :)
Let's discuss a one-off way to copy data over for tests.
Is there another another way than through public-endpot?
If not, we shold start the copy :)

Change 478962 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix comment in cloud-analytics common.yaml

https://gerrit.wikimedia.org/r/478962

Change 478963 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add cloud_analytics roles

https://gerrit.wikimedia.org/r/478963

Change 478962 merged by Ottomata:
[operations/puppet@production] Fix comment in cloud-analytics common.yaml

https://gerrit.wikimedia.org/r/478962

Change 478963 merged by Ottomata:
[operations/puppet@production] Add cloud_analytics roles

https://gerrit.wikimedia.org/r/478963

Change 468070 abandoned by Ottomata:
[WIP] Configure cloud-analytics-eqiad Hadoop cluster

Reason:
doing this in Cloud VPS

https://gerrit.wikimedia.org/r/468070

Change 479324 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use fqdn instead of hostname in nodes id

https://gerrit.wikimedia.org/r/479324

Change 479324 merged by Ottomata:
[operations/puppet@production] Use fqdn instead of hostname in nodes id

https://gerrit.wikimedia.org/r/479324

Change 479339 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Set presto hive connector name to hive-hadoop2

https://gerrit.wikimedia.org/r/479339

Change 479339 merged by Ottomata:
[operations/puppet@production] Set presto hive connector name to hive-hadoop2

https://gerrit.wikimedia.org/r/479339

Change 479343 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use thrift:// uri for hive http:// uri for discovery uri

https://gerrit.wikimedia.org/r/479343

Change 479343 merged by Ottomata:
[operations/puppet@production] Use thrift:// uri for hive http:// uri for discovery uri

https://gerrit.wikimedia.org/r/479343

Change 479508 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Run Presto coordinator on ca-master-2 in cloud-analytics

https://gerrit.wikimedia.org/r/479508

Change 479508 merged by Ottomata:
[operations/puppet@production] Run Presto coordinator on ca-master-2 in cloud-analytics

https://gerrit.wikimedia.org/r/479508

Change 479509 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add ::profile::presto::server to ca-master-2 to run Presto coordinator

https://gerrit.wikimedia.org/r/479509

Change 479509 merged by Ottomata:
[operations/puppet@production] Add ::profile::presto::server to ca-master-2 to run Presto coordinator

https://gerrit.wikimedia.org/r/479509

Change 479525 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Make presto module and profile smarter

https://gerrit.wikimedia.org/r/479525

Change 479525 merged by Ottomata:
[operations/puppet@production] Make presto module and profile smarter

https://gerrit.wikimedia.org/r/479525

Change 480243 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix Hadoop settings for presto in cloud-analytics

https://gerrit.wikimedia.org/r/480243

Change 480243 merged by Ottomata:
[operations/puppet@production] Fix Hadoop settings for presto in cloud-analytics

https://gerrit.wikimedia.org/r/480243

Ottomata set the point value for this task to 13.
Ottomata moved this task from In Code Review to Done on the Analytics-Kanban board.Jan 8 2019, 4:11 PM
Nuria added a comment.Feb 14 2019, 5:26 AM

After our talk with ops and cloud looks like we are going to need to move this work to prod due to lack of monitoring in labs. Project blocked

Nuria moved this task from Done to Paused on the Analytics-Kanban board.Feb 14 2019, 5:26 AM

If cloudelastic can do this, so can we!