Page MenuHomePhabricator

Create new Hive / Oozie server from old analytics Dell {hawk}
Closed, ResolvedPublic13 Story Points

Description

Now that most of the new nodes have been added to the cluster (we are only waiting on 2 more), we have some room to repurpose a few of the older nodes. First up, I'd like to repurpose one of these as the Hive / Oozie server.

This involves:

  • Decommission an old Hadoop node
  • Reinstall node.
  • Stop Oozie, Hive, MySQL move to newly installed node, and puppetize.
  • Make sure MySQL DB is properly backed up.
  • Adjust Hive heapsizes in role::analytics::hive::server

Event Timeline

Ottomata created this task.Aug 24 2015, 6:38 PM
Ottomata claimed this task.
Ottomata raised the priority of this task from to Normal.
Ottomata updated the task description. (Show Details)

I have excluded analytics1015 from active Hadoop nodes. Waiting for data to be moved elsewhere.

Change 238479 had a related patch set uploaded (by Ottomata):
Use raid1-lvm-ext4.cfg for analytics1015

https://gerrit.wikimedia.org/r/238479

Change 238479 merged by Ottomata:
Use raid1-lvm-ext4.cfg for analytics1015

https://gerrit.wikimedia.org/r/238479

Change 238508 had a related patch set uploaded (by Ottomata):
Puppetize MySQL/MariaDB server on analytics1015 in prep for moving Hive and Oozie

https://gerrit.wikimedia.org/r/238508

Change 238508 merged by Ottomata:
Puppetize MySQL/MariaDB server on analytics1015 in prep for moving Hive and Oozie

https://gerrit.wikimedia.org/r/238508

Change 238529 had a related patch set uploaded (by Ottomata):
Puppetize MariaDB init.d and bin in role::analytics::mysql::meta

https://gerrit.wikimedia.org/r/238529

Change 238529 merged by Ottomata:
Puppetize MariaDB init.d and bin in role::analytics::mysql::meta

https://gerrit.wikimedia.org/r/238529

Change 238832 had a related patch set uploaded (by Ottomata):
Enable error logging for Hive/Oozie meta mysql, set binlog_format=ROW

https://gerrit.wikimedia.org/r/238832

Change 238832 merged by Ottomata:
Enable error logging for Hive/Oozie meta mysql, set binlog_format=ROW

https://gerrit.wikimedia.org/r/238832

Ottomata updated the task description. (Show Details)Sep 16 2015, 7:57 PM
Ottomata set Security to None.
Milimetric renamed this task from Create new Hive / Oozie server from old analytics Dell to Create new Hive / Oozie server from old analytics Dell {hawk}.Sep 24 2015, 3:49 PM

Change 264742 had a related patch set uploaded (by Ottomata):
Use hiera to configure hive and oozie server hostnames

https://gerrit.wikimedia.org/r/264742

Change 264742 merged by Ottomata:
Use hiera to configure hive and oozie server hostnames

https://gerrit.wikimedia.org/r/264742

Ottomata moved this task from Paused to In Progress on the Analytics-Kanban board.Jan 18 2016, 5:08 PM

Testing out 'Stop Oozie, Hive, MySQL move to newly installed node, and puppetize.' in labs. Prod Steps are:

  1. suspend all oozie (hive) jobs.
  2. stop puppet on analytics1027.
  3. stop oozie, hive, mysql on analytics1027.
  4. stop mysql slave on analytics1015. Make sure it is able to be a master (readonly = false?)
  5. Merge this puppet change: https://gerrit.wikimedia.org/r/#/c/264760/
  6. apply puppet on analytics1027, analytics1015, stat1002 (and eventually everywhere)
  7. Make sure analytics1015 is now running oozie, hive-server2 and hive-metastore
  8. Make sure stat1002 can talk to oozie and hive-server2 like normal.
  9. On analytics1027: apt-get remove hive-server2 hive-metastore oozie mysql-server
Ottomata updated the task description. (Show Details)Jan 18 2016, 5:29 PM
Ottomata renamed this task from Create new Hive / Oozie server from old analytics Dell {hawk} to Create new Hive / Oozie server from old analytics Dell {hawk} [13 pts].Jan 18 2016, 5:55 PM
Ottomata moved this task from In Progress to Paused on the Analytics-Kanban board.Jan 27 2016, 5:14 PM
Milimetric renamed this task from Create new Hive / Oozie server from old analytics Dell {hawk} [13 pts] to Create new Hive / Oozie server from old analytics Dell {hawk}.Feb 22 2016, 9:06 PM
Milimetric set the point value for this task to 13.

Change 272605 had a related patch set uploaded (by Ottomata):
Move Hive and Oozie to analytics1015

https://gerrit.wikimedia.org/r/272605

Change 272606 had a related patch set uploaded (by Ottomata):
Make MySQL instance on analytics1015 the master

https://gerrit.wikimedia.org/r/272606

Change 272606 merged by Ottomata:
Make MySQL instance on analytics1015 the master

https://gerrit.wikimedia.org/r/272606

Change 272605 merged by Ottomata:
Move Hive and Oozie to analytics1015

https://gerrit.wikimedia.org/r/272605

Change 272783 had a related patch set uploaded (by Joal):
Update hiera heapsize for hive and oozie servers

https://gerrit.wikimedia.org/r/272783

Change 272783 merged by Ottomata:
Update hiera heapsize for hive and oozie servers

https://gerrit.wikimedia.org/r/272783

Ottomata moved this task from Paused to Done on the Analytics-Kanban board.Feb 23 2016, 9:45 PM
Nuria closed this task as Resolved.Mar 8 2016, 9:07 PM