Page MenuHomePhabricator

Apply Xms Java Heap settings to all the Hadoop daemons
Closed, ResolvedPublic

Description

In T157968 we specified -Xms and -Xmx settings for the Zookeeper JVMs and we observed a positive impact to the GC behavior. This positive impact was due to the Xms value set equal to the Xmx one, forcing the minimum heap size allocated (if not specified it is the default, 256m) to max heap right from the start of the JVM. This also fixes the value of Heap memory committed to max heap, allowing the GC to be less conservative. More details documented in: https://wikitech.wikimedia.org/wiki/JVM_Tuning#Setting_max_and_min_heap_.28-Xmx_and_-Xms.29

I believe that specifying the Xmx settings equal to Xmx for most of the Hadoop JVMs (Yarn RM and NM, HDFS NN and DN) will have a positive impact.

Event Timeline

elukey moved this task from Backlog to In Progress on the User-Elukey board.
elukey moved this task from In Progress to Analytics Backlog on the User-Elukey board.

Change 346983 had a related patch set uploaded (by Elukey):
[operations/puppet@production] Add explicit Xmx and Xms settings to Hadoop MRHS and Namenode

https://gerrit.wikimedia.org/r/346983

Change 346983 merged by Elukey:
[operations/puppet@production] Add explicit Xmx and Xms settings to Hadoop MRHS and Namenode

https://gerrit.wikimedia.org/r/346983

List of daemons that could benefit from Xms:

  • Hadoop HDFS datanode (workers)
  • Hadoop Yarn nodemanager (workers)
  • Hadoop Yarn Resource Manager (master nodes)

Daemons already done:

  • Hadoop MapReduce History server (active master node)
  • Hadoop Nameserver (master nodes)

Change 347353 had a related patch set uploaded (by Elukey):
[operations/puppet/cdh@master] Add JVM options tunables for Yarn RM and Hadoop DN/NN

https://gerrit.wikimedia.org/r/347353

Change 347353 merged by Elukey:
[operations/puppet/cdh@master] Add JVM options tunables for Yarn RM and Hadoop DN/NN

https://gerrit.wikimedia.org/r/347353

Change 347814 had a related patch set uploaded (by Elukey):
[operations/puppet@production] Set Xms settings for Hadoop Workers daemons

https://gerrit.wikimedia.org/r/347814

Change 347814 merged by Elukey:
[operations/puppet@production] Set Xms settings for Hadoop Workers daemons

https://gerrit.wikimedia.org/r/347814

Change 348915 had a related patch set uploaded (by Elukey):
[operations/puppet@production] Set Xms value for the Hadoop Yarn Resource Manager's JVM

https://gerrit.wikimedia.org/r/348915

I can see positive effects on Datanodes and Nodemanagers after a week of data-points. The heap used is more of course (because of the higher Xms threshold) but overall the GC collection overhead seems to have gone down and stabilized.

Screen Shot 2017-04-19 at 10.51.08 AM.png (600×1 px, 555 KB)

Screen Shot 2017-04-19 at 10.52.24 AM.png (816×1 px, 706 KB)

Last one to do is the Yarn Resource Manager and the task will be done (this will need to wait the MW switchover happening today).

Change 348915 merged by Elukey:
[operations/puppet@production] Set Xms value for the Hadoop Yarn Resource Manager's JVM

https://gerrit.wikimedia.org/r/348915

Applied settings to the Yarn Resource Manager daemons.