Page MenuHomePhabricator
Create Task
Maniphest T159219

Apply Xms Java Heap settings to all the Hadoop daemons
Closed, ResolvedPublic
Actions

Description

In T157968 we specified -Xms and -Xmx settings for the Zookeeper JVMs and we observed a positive impact to the GC behavior. This positive impact was due to the Xms value set equal to the Xmx one, forcing the minimum heap size allocated (if not specified it is the default, 256m) to max heap right from the start of the JVM. This also fixes the value of Heap memory committed to max heap, allowing the GC to be less conservative. More details documented in: https://wikitech.wikimedia.org/wiki/JVM_Tuning#Setting_max_and_min_heap_.28-Xmx_and_-Xms.29

I believe that specifying the Xmx settings equal to Xmx for most of the Hadoop JVMs (Yarn RM and NM, HDFS NN and DN) will have a positive impact.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Set Xms value for the Hadoop Yarn Resource Manager's JVMoperations/puppetproduction+1 -0
Set Xms settings for Hadoop Workers daemonsoperations/puppetproduction+3 -1
Add JVM options tunables for Yarn RM and Hadoop DN/NNoperations/puppet/cdhmaster+20 -5
Add explicit Xmx and Xms settings to Hadoop MRHS and Namenodeoperations/puppetproduction+4 -1
Customize query in gerrit

Related Objects

Event Timeline

elukey created this task.Feb 28 2017, 11:41 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 28 2017, 11:41 AM
• Nuria moved this task from Incoming to Wikistats on the Analytics board.Mar 2 2017, 5:10 PM
elukey added a project: User-Elukey.Mar 14 2017, 7:15 AM
elukey moved this task from Backlog to In Progress on the User-Elukey board.
elukey moved this task from In Progress to Analytics Backlog on the User-Elukey board.
gerritbot subscribed.Apr 7 2017, 1:43 PM

Change 346983 had a related patch set uploaded (by Elukey):
[operations/puppet@production] Add explicit Xmx and Xms settings to Hadoop MRHS and Namenode

https://gerrit.wikimedia.org/r/346983

gerritbot added a project: Patch-For-Review.Apr 7 2017, 1:43 PM
gerritbot added a comment.Apr 7 2017, 2:02 PM

Change 346983 merged by Elukey:
[operations/puppet@production] Add explicit Xmx and Xms settings to Hadoop MRHS and Namenode

https://gerrit.wikimedia.org/r/346983

elukey edited projects, added Analytics-Kanban; removed Patch-For-Review, Analytics-Clusters, Analytics.Apr 7 2017, 2:35 PM
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.
elukey added a comment.Apr 7 2017, 2:45 PM

List of daemons that could benefit from Xms:

  • Hadoop HDFS datanode (workers)
  • Hadoop Yarn nodemanager (workers)
  • Hadoop Yarn Resource Manager (master nodes)

Daemons already done:

  • Hadoop MapReduce History server (active master node)
  • Hadoop Nameserver (master nodes)
elukey added a comment.Apr 7 2017, 3:48 PM

https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-common/ClusterSetup.html

gerritbot added a comment.Apr 10 2017, 11:31 AM

Change 347353 had a related patch set uploaded (by Elukey):
[operations/puppet/cdh@master] Add JVM options tunables for Yarn RM and Hadoop DN/NN

https://gerrit.wikimedia.org/r/347353

gerritbot added a project: Patch-For-Review.Apr 10 2017, 11:31 AM
gerritbot added a comment.Apr 12 2017, 7:56 AM

Change 347353 merged by Elukey:
[operations/puppet/cdh@master] Add JVM options tunables for Yarn RM and Hadoop DN/NN

https://gerrit.wikimedia.org/r/347353

gerritbot added a comment.Apr 12 2017, 8:23 AM

Change 347814 had a related patch set uploaded (by Elukey):
[operations/puppet@production] Set Xms settings for Hadoop Workers daemons

https://gerrit.wikimedia.org/r/347814

gerritbot added a comment.Apr 12 2017, 8:51 AM

Change 347814 merged by Elukey:
[operations/puppet@production] Set Xms settings for Hadoop Workers daemons

https://gerrit.wikimedia.org/r/347814

• Nuria moved this task from In Progress to In Code Review on the Analytics-Kanban board.Apr 12 2017, 3:14 PM
• Nuria moved this task from In Code Review to In Progress on the Analytics-Kanban board.Apr 13 2017, 3:08 PM
gerritbot added a comment.Apr 19 2017, 8:43 AM

Change 348915 had a related patch set uploaded (by Elukey):
[operations/puppet@production] Set Xms value for the Hadoop Yarn Resource Manager's JVM

https://gerrit.wikimedia.org/r/348915

elukey added a comment.EditedApr 19 2017, 8:55 AM

I can see positive effects on Datanodes and Nodemanagers after a week of data-points. The heap used is more of course (because of the higher Xms threshold) but overall the GC collection overhead seems to have gone down and stabilized.

Screen Shot 2017-04-19 at 10.51.08 AM.png (600×1 px, 555 KB)

Screen Shot 2017-04-19 at 10.52.24 AM.png (816×1 px, 706 KB)

Last one to do is the Yarn Resource Manager and the task will be done (this will need to wait the MW switchover happening today).

elukey moved this task from In Progress to In Code Review on the Analytics-Kanban board.Apr 19 2017, 10:48 AM
gerritbot added a comment.Apr 20 2017, 9:54 AM

Change 348915 merged by Elukey:
[operations/puppet@production] Set Xms value for the Hadoop Yarn Resource Manager's JVM

https://gerrit.wikimedia.org/r/348915

elukey moved this task from In Code Review to Done on the Analytics-Kanban board.Apr 20 2017, 12:46 PM

Applied settings to the Yarn Resource Manager daemons.

• Nuria closed this task as Resolved.Apr 21 2017, 6:46 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits