Page MenuHomePhabricator
Create Task
Maniphest T255139

Create the new Hadoop test cluster
Closed, ResolvedPublic
Actions

Description

High level things to do:

  1. Help dcops to rack/setup/deploy the new hosts: two masters, one coordinator and three workers. This will likely need some partman config or similar.
  2. Come up with the the new config in puppet for the test cluster. I have gathered some things to keep in mind when bootstrapping a testing cluster https://wikitech.wikimedia.org/wiki/User:Elukey/Analytics/Hadoop
  3. Once all is done, decommission the old test cluster and give the green light to the DC ops team to decommission analytics1028->40.
  4. Special consideration for analytics1041 that runs our Druid testing cluster. We'll need to find a solution, maybe a co-location somewhere or possibly a VM (more likely). 1041 will need to be decommed as well.

See subtasks for more info :)

Details

SubjectRepoBranchLines +/-
Decommission an-tool1006operations/puppetproduction+0 -12
Add role::druid::test_analytics::worker to an-test-druid1001operations/puppetproduction+1 -1
Review settings for the new Druid test clusteroperations/puppetproduction+13 -14
Move upload of hive-site.xml from client to hadoop standby in testoperations/puppetproduction+8 -2
Add analytics_test_cluster::client role to an-test-client1001operations/puppetproduction+1 -1
role::analytics_test_cluster::client: upload hive-site.xmloperations/puppetproduction+4 -0
profile::hadoop::master: refactor monitor for HDFS space leftoperations/puppetproduction+6 -6
Add an-test-coord1001's IPs to Kafka Jumbo's ferm rulesoperations/puppetproduction+2 -0
Add IPv6's PTR/AAAA records for the new Hadoop test clusteroperations/dnsmaster+12 -0
profile::hadoop::worker: avoid custom disk space checks for the test clusteroperations/puppetproduction+16 -11
Add fake keytabs for new Hadoop test cluster nodeslabs/privatemaster+0 -0
role::analytics_test_cluster::hadoop::worker: remove unnecessary nrpe disk checkoperations/puppetproduction+0 -5
role::analytics_test_cluster::coordinator: set /srv/mysql as datadir for mariadboperations/puppetproduction+1 -0
Reduce the HDFS block replication factor for the Hadoop test clusteroperations/puppetproduction+3 -0
role::analytics_test_cluster::coordiantor: avoid db backupsoperations/puppetproduction+0 -7
Set the test coordinator role to an-test-coord1001operations/puppetproduction+1 -1
Fix typos for Hadoop test cluster's hostnamesoperations/puppetproduction+10 -10
Avoid the analytics keytab for the Analytics Hadoop test masteroperations/puppetproduction+0 -5
Remove min disk available constraint from Hadoop test workers' settingsoperations/puppetproduction+0 -5
Set up the new Analytics Hadoop test clusteroperations/puppetproduction+13 -9
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

elukey created this task.Jun 11 2020, 1:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 11 2020, 1:32 PM
elukey edited projects, added Analytics-Clusters; removed Analytics-Kanban.Jun 11 2020, 1:33 PM
elukey moved this task from Backlog to Q1 2020/2021 on the Analytics-Clusters board.
• fdans moved this task from Incoming to Operational Excellence on the Analytics board.Jun 15 2020, 3:54 PM
Aklapper removed a project: Analytics.Jul 4 2020, 7:59 AM
elukey added subtasks: T255520: (Due By: 2020-07-17) rack/setup/install <an-test-worker1001-1003>, T255518: (Due By: 2020-07-02) rack/setup/install 3 lightweight hadoop nodes.Aug 14 2020, 9:40 AM
elukey added a subtask: T227485: Decommission analytics10[28-31,33-41].Aug 14 2020, 9:43 AM
elukey triaged this task as Medium priority.Aug 14 2020, 9:47 AM
elukey updated the task description. (Show Details)
elukey added a comment.Aug 27 2020, 7:25 AM

The new hosts are ready to be used:

  • an-test-master100[1,2] will be the Hadoop master nodes
  • an-test-coord1001 will be the coordinator/launcher node
  • an-test-worker100[1-3] will be the Hadoop worker nodes

Some notes to think about:

  • The new Hadoop workers have a single RAID10 lvs volume, and the majority of the disk space is under /srv. So we'll have only one directory listed in puppet/hiera for datanode dirs. The current hadoop test cluster is more similar to the production one, in which we have 12 disks on each node (without any raid) and each of them has a separate partition, that is listed in puppet.
  • analytics1041 and analytics1039 are special nodes, namely Druid test and Hue test. Since the hosts will need to be decommed we'll need probably to create some little VMs for Druid/Hue/etc..
• Cmjohnson closed subtask T255518: (Due By: 2020-07-02) rack/setup/install 3 lightweight hadoop nodes as Resolved.Aug 27 2020, 1:36 PM
• Cmjohnson closed subtask T255520: (Due By: 2020-07-17) rack/setup/install <an-test-worker1001-1003> as Resolved.
elukey changed the status of subtask T227485: Decommission analytics10[28-31,33-41] from Stalled to Open.Sep 28 2020, 9:17 AM
elukey moved this task from Q1 2020/2021 to Q1 2021/2022 on the Analytics-Clusters board.Oct 6 2020, 12:39 PM
• Cmjohnson closed subtask T227485: Decommission analytics10[28-31,33-41] as Resolved.Oct 7 2020, 2:58 PM
elukey claimed this task.Oct 9 2020, 10:38 AM
elukey added a project: Analytics-Kanban.
gerritbot added a comment.Oct 9 2020, 10:39 AM

Change 633162 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Set up the new Analytics Hadoop test cluster

https://gerrit.wikimedia.org/r/633162

gerritbot added a project: Patch-For-Review.Oct 9 2020, 10:39 AM
gerritbot added a comment.Oct 9 2020, 10:58 AM

Change 633162 merged by Elukey:
[operations/puppet@production] Set up the new Analytics Hadoop test cluster

https://gerrit.wikimedia.org/r/633162

gerritbot added a comment.Oct 9 2020, 11:02 AM

Change 633164 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Remove min disk available constraint from Hadoop test workers' settings

https://gerrit.wikimedia.org/r/633164

gerritbot added a comment.Oct 9 2020, 11:04 AM

Change 633164 merged by Elukey:
[operations/puppet@production] Remove min disk available constraint from Hadoop test workers' settings

https://gerrit.wikimedia.org/r/633164

Maintenance_bot removed a project: Patch-For-Review.Oct 9 2020, 11:10 AM
gerritbot added a comment.Oct 9 2020, 11:43 AM

Change 633168 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Avoid the analytics keytab for the Analytics Hadoop test master

https://gerrit.wikimedia.org/r/633168

gerritbot added a project: Patch-For-Review.Oct 9 2020, 11:43 AM

Change 633168 merged by Elukey:
[operations/puppet@production] Avoid the analytics keytab for the Analytics Hadoop test master

https://gerrit.wikimedia.org/r/633168

Maintenance_bot removed a project: Patch-For-Review.Oct 9 2020, 12:10 PM
gerritbot added a comment.Oct 9 2020, 1:43 PM

Change 633187 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Fix typos for Hadoop test cluster's hostnames

https://gerrit.wikimedia.org/r/633187

gerritbot added a project: Patch-For-Review.Oct 9 2020, 1:43 PM

Change 633187 merged by Elukey:
[operations/puppet@production] Fix typos for Hadoop test cluster's hostnames

https://gerrit.wikimedia.org/r/633187

Maintenance_bot removed a project: Patch-For-Review.Oct 9 2020, 2:10 PM
gerritbot added a comment.Oct 12 2020, 5:54 AM

Change 633386 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Set the test coordinator role to an-test-coord1001

https://gerrit.wikimedia.org/r/633386

gerritbot added a project: Patch-For-Review.Oct 12 2020, 5:54 AM

Change 633386 merged by Elukey:
[operations/puppet@production] Set the test coordinator role to an-test-coord1001

https://gerrit.wikimedia.org/r/633386

Maintenance_bot removed a project: Patch-For-Review.Oct 12 2020, 6:10 AM
gerritbot added a comment.Oct 12 2020, 6:32 AM

Change 633387 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_test_cluster::coordiantor: avoid db backups

https://gerrit.wikimedia.org/r/633387

gerritbot added a project: Patch-For-Review.Oct 12 2020, 6:32 AM

Change 633387 merged by Elukey:
[operations/puppet@production] role::analytics_test_cluster::coordiantor: avoid db backups

https://gerrit.wikimedia.org/r/633387

gerritbot added a comment.Oct 12 2020, 6:53 AM

Change 633471 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Reduce the HDFS block replication factor for the Hadoop test cluster

https://gerrit.wikimedia.org/r/633471

gerritbot added a comment.Oct 12 2020, 6:55 AM

Change 633471 merged by Elukey:
[operations/puppet@production] Reduce the HDFS block replication factor for the Hadoop test cluster

https://gerrit.wikimedia.org/r/633471

gerritbot added a comment.Oct 12 2020, 7:57 AM

Change 633476 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_test_cluster::coordinator: set /srv/mysql as datadir for mariadb

https://gerrit.wikimedia.org/r/633476

gerritbot added a comment.Oct 12 2020, 7:58 AM

Change 633476 merged by Elukey:
[operations/puppet@production] role::analytics_test_cluster::coordinator: set /srv/mysql as datadir for mariadb

https://gerrit.wikimedia.org/r/633476

elukey added subscribers: klausman, • razzi.Oct 12 2020, 8:54 AM

The basic test cluster is set up:

an-master100[1,2] - Hadoop masters
an-coord1001 - test coordinator
an-worker100[1-3] - workers

The total HDFS space is around 42T (not a lot), with HDFS block replication factor 2 (not 3 as the main cluster). A couple of things are missing:

  • an-tool1006 (test client) is running Stretch, meanwhile all our stat100x are on Buster, so we should decom that VM and create a new one with Buster.
  • analytics1041 was the test host for Druid, so we should probably create a VM for it.
  • Hue is missing, so probably another little VM is also needed too.

@razzi @klausman the above I think is a good set of little tasks to get familiar with Hadoop and Ganeti (the infra that we use to run VMs - https://wikitech.wikimedia.org/wiki/Ganeti). Let's schedule the work later on, and let me know if there is anything that you'd prefer to do.

gerritbot added a comment.Oct 14 2020, 7:11 AM

Change 633944 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_test_cluster::hadoop::worker: remove unnecessary nrpe disk check

https://gerrit.wikimedia.org/r/633944

gerritbot added a comment.Oct 14 2020, 7:12 AM

Change 633944 merged by Elukey:
[operations/puppet@production] role::analytics_test_cluster::hadoop::worker: remove unnecessary nrpe disk check

https://gerrit.wikimedia.org/r/633944

gerritbot added a comment.Oct 14 2020, 7:18 AM

Change 633945 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add an-test-coord1001's IPs to Kafka Jumbo's ferm rules

https://gerrit.wikimedia.org/r/633945

gerritbot added a comment.Oct 14 2020, 7:34 AM

Change 633946 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/dns@master] Add IPv6's PTR/AAAA records for the new Hadoop test cluster

https://gerrit.wikimedia.org/r/633946

gerritbot added a comment.Oct 14 2020, 8:05 AM

Change 633947 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hadoop::worker: avoid custom disk space checks for the test cluster

https://gerrit.wikimedia.org/r/633947

gerritbot added a comment.Oct 14 2020, 8:16 AM

Change 633948 had a related patch set uploaded (by Elukey; owner: Elukey):
[labs/private@master] Add fake keytabs for new Hadoop test cluster nodes

https://gerrit.wikimedia.org/r/633948

gerritbot added a comment.Oct 14 2020, 8:16 AM

Change 633948 merged by Elukey:
[labs/private@master] Add fake keytabs for new Hadoop test cluster nodes

https://gerrit.wikimedia.org/r/633948

gerritbot added a comment.Oct 14 2020, 8:17 AM

Change 633947 merged by Elukey:
[operations/puppet@production] profile::hadoop::worker: avoid custom disk space checks for the test cluster

https://gerrit.wikimedia.org/r/633947

gerritbot added a comment.Oct 14 2020, 8:36 AM

Change 633946 merged by Elukey:
[operations/dns@master] Add IPv6's PTR/AAAA records for the new Hadoop test cluster

https://gerrit.wikimedia.org/r/633946

elukey mentioned this in rLPRIce3e2d8e2b28: Add fake keytabs for new Hadoop test cluster nodes.Oct 14 2020, 8:36 AM
gerritbot added a comment.Oct 14 2020, 8:37 AM

Change 633945 merged by Elukey:
[operations/puppet@production] Add an-test-coord1001's IPs to Kafka Jumbo's ferm rules

https://gerrit.wikimedia.org/r/633945

gerritbot added a comment.Oct 14 2020, 10:44 AM

Change 633973 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] profile::hadoop::master: refactor monitor for HDFS space left

https://gerrit.wikimedia.org/r/633973

gerritbot added a comment.Oct 14 2020, 10:50 AM

Change 633973 merged by Elukey:
[operations/puppet@production] profile::hadoop::master: refactor monitor for HDFS space left

https://gerrit.wikimedia.org/r/633973

elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.Oct 14 2020, 4:16 PM
gerritbot added a comment.Oct 21 2020, 3:14 PM

Change 635563 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::analytics_test_cluster::client: uplaad hive-site.xml

https://gerrit.wikimedia.org/r/635563

gerritbot added a comment.Oct 21 2020, 3:17 PM

Change 635563 merged by Elukey:
[operations/puppet@production] role::analytics_test_cluster::client: upload hive-site.xml

https://gerrit.wikimedia.org/r/635563

gerritbot added a comment.Oct 21 2020, 10:38 PM

Change 635662 had a related patch set uploaded (by Razzi; owner: Razzi):
[operations/puppet@production] Add analytics_test_cluster::client role to an-test-client1001

https://gerrit.wikimedia.org/r/635662

gerritbot added a comment.Oct 21 2020, 10:51 PM

Change 635662 merged by Razzi:
[operations/puppet@production] Add analytics_test_cluster::client role to an-test-client1001

https://gerrit.wikimedia.org/r/635662

elukey added a comment.Oct 22 2020, 6:58 AM

Created all the webrequest tables from the hive scripts in refinery.

elukey added a subtask: T266064: Site: 1 VM request for Analytics test cluster.Oct 22 2020, 8:52 AM
elukey added a comment.Oct 22 2020, 9:00 AM

Remaining steps:

  • Replace an-tool1006 with a buster VM (already in progress)
  • Create a new VM for Druid
  • Create a new VM for Hue
gerritbot added a comment.Oct 23 2020, 9:33 AM

Change 635959 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Move upload of hive-site.xml from client to hadoop standby in test

https://gerrit.wikimedia.org/r/635959

gerritbot added a comment.Oct 23 2020, 9:34 AM

Change 635959 merged by Elukey:
[operations/puppet@production] Move upload of hive-site.xml from client to hadoop standby in test

https://gerrit.wikimedia.org/r/635959

elukey closed subtask T266064: Site: 1 VM request for Analytics test cluster as Resolved.Oct 27 2020, 8:04 AM
gerritbot added a comment.Oct 29 2020, 2:34 PM

Change 637502 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Review settings for the new Druid test cluster

https://gerrit.wikimedia.org/r/637502

gerritbot added a comment.Oct 29 2020, 2:38 PM

Change 637502 merged by Elukey:
[operations/puppet@production] Review settings for the new Druid test cluster

https://gerrit.wikimedia.org/r/637502

gerritbot added a comment.Oct 29 2020, 4:09 PM

Change 637526 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add role::druid::test_analytics::worker to an-test-druid1001

https://gerrit.wikimedia.org/r/637526

gerritbot added a comment.Oct 29 2020, 4:30 PM

Change 637526 merged by Elukey:
[operations/puppet@production] Add role::druid::test_analytics::worker to an-test-druid1001

https://gerrit.wikimedia.org/r/637526

elukey added a comment.Oct 30 2020, 8:46 AM

Created https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Test

Bootstrapped Hue/Yarn and Druid, I think we are now good!

The last step is to decommission an-tool1006 and then we are done.

Stashbot added a comment.Oct 30 2020, 8:54 AM

Mentioned in SAL (#wikimedia-operations) [2020-10-30T08:54:11Z] <elukey> decom an-tool1006 (old analytics test vm) - T255139

gerritbot added a comment.Oct 30 2020, 8:55 AM

Change 637638 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Decommission an-tool1006

https://gerrit.wikimedia.org/r/637638

ops-monitoring-bot added a comment.Oct 30 2020, 8:58 AM

cookbooks.sre.hosts.decommission executed by elukey@cumin1001 for hosts: an-tool1006.eqiad.wmnet

  • an-tool1006.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
  • COMMON_STEPS (FAIL)
    • Failed to run the sre.dns.netbox cookbook: Cumin execution failed (exit_code=2)

ERROR: some step on some host failed, check the bolded items above

gerritbot added a comment.Oct 30 2020, 9:05 AM

Change 637638 merged by Elukey:
[operations/puppet@production] Decommission an-tool1006

https://gerrit.wikimedia.org/r/637638

elukey moved this task from In Progress to Done on the Analytics-Kanban board.Oct 30 2020, 9:15 AM
elukey moved this task from Q1 2021/2022 to Done on the Analytics-Clusters board.Nov 2 2020, 10:11 AM
• fdans closed this task as Resolved.Nov 16 2020, 3:51 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits