We don't have metrics from the historyserver in graphite. Write jmxtrans configs to get these.
Description
Description
Details
Details
Customize query in gerrit
Event Timeline
Comment Actions
I think that the history server does not have any jmx port configured, and I didn't find any environment variable to put in hadoop-env.sh to fix it. I am also fairly ignorant about configs in CDH, I tried to check the docs but nothing really good came up.
@Ottomata - any suggestion/hint? :)
Comment Actions
Ah! I was looking in the wrong place!
From https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/ClusterSetup.html#Configuring_the_Hadoop_Daemons I can see HADOOP_JOB_HISTORYSERVER_OPTS
Comment Actions
Change 334667 had a related patch set uploaded (by Elukey):
Add JMX port 9986 to the MapReduce History process
Comment Actions
Next step is to understand if we can expect metrics exported via JMX from the JobHistory..
Comment Actions
Change 339418 had a related patch set uploaded (by Elukey):
Update the cdh submodule to the latest sha
Comment Actions
Restarted the history server daemon on an1001, all good, port 9986 is now used by JMX. I verified via jconsole that the JVM MBeans are there, so the next step is to add the necessary jmxtrans configuration to start pushing metrics to graphite.
Comment Actions
Change 342592 had a related patch set uploaded (by Elukey):
[operations/puppet/cdh] Add the Jmxtrans configuration for the MapReduce History server
Comment Actions
Change 342592 merged by Elukey:
[operations/puppet/cdh] Add the Jmxtrans configuration for the MapReduce History server
Comment Actions
Change 342640 had a related patch set uploaded (by Elukey):
[operations/puppet] Update the cdh module's sha with the latest change
Comment Actions
Change 342640 merged by Elukey:
[operations/puppet] Update the cdh module's sha with the latest change
Comment Actions
Metrics are flowing to graphite, added graphs to https://grafana.wikimedia.org/dashboard/db/analytics-hadoop