I tried jmx_exporter on deployment-logstash2 with the results below. A few notes: the exporter config needs to be somewhere accessible by elasticsearch (e.g. /srv/elasticsearch) or asking for metrics fails with something like

<title>Error 500 access denied ("java.io.FilePermission"  "/etc/prometheus/jmx_exporter.yaml" "read")</title>

In later versions of jmx_exporter the permission denied error isn't even reported and jetty just closes the connection without a status code.
Also the scrape is marked as failed by jmx_exporter though some metrics _have_ been scraped.

$ curl localhost:9499/metrics  -s
# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.
# TYPE jmx_scrape_duration_seconds gauge
jmx_scrape_duration_seconds 0.001576552
# HELP jmx_scrape_error Non-zero if this scrape failed.
# TYPE jmx_scrape_error gauge
jmx_scrape_error 1.0
# HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
# TYPE jvm_gc_collection_seconds summary
jvm_gc_collection_seconds_count{gc="PS Scavenge",} 12.0
jvm_gc_collection_seconds_sum{gc="PS Scavenge",} 0.945
jvm_gc_collection_seconds_count{gc="PS MarkSweep",} 3.0
jvm_gc_collection_seconds_sum{gc="PS MarkSweep",} 0.414
# HELP jmx_config_reload_failure_total Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_total counter
jmx_config_reload_failure_total 0.0
# HELP jvm_threads_current Current thread count of a JVM
# TYPE jvm_threads_current gauge
jvm_threads_current 77.0
# HELP jvm_threads_daemon Daemon thread count of a JVM
# TYPE jvm_threads_daemon gauge
jvm_threads_daemon 69.0
# HELP jvm_threads_peak Peak thread count of a JVM
# TYPE jvm_threads_peak gauge
jvm_threads_peak 89.0
# HELP jvm_threads_started_total Started thread count of a JVM
# TYPE jvm_threads_started_total counter
jvm_threads_started_total 122.0
# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
# TYPE jvm_threads_deadlocked gauge
jvm_threads_deadlocked 0.0
# HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
# TYPE jvm_threads_deadlocked_monitor gauge
jvm_threads_deadlocked_monitor 0.0
# HELP jvm_info JVM version info
# TYPE jvm_info gauge
jvm_info{version="1.8.0_151-8u151-b12-1~bpo8+1-b12",vendor="Oracle Corporation",} 1.0
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.512145079031E9
# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 11670.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 11672.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 2.0
# HELP jmx_config_reload_success_total Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_total counter
jmx_config_reload_success_total 0.0
# HELP jvm_memory_bytes_used Used bytes of a given JVM memory area.
# TYPE jvm_memory_bytes_used gauge
jvm_memory_bytes_used{area="heap",} 1.51883844E9
jvm_memory_bytes_used{area="nonheap",} 1.06190416E8
# HELP jvm_memory_bytes_committed Committed (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_committed gauge
jvm_memory_bytes_committed{area="heap",} 5.00170752E9
jvm_memory_bytes_committed{area="nonheap",} 1.11788032E8
# HELP jvm_memory_bytes_max Max (bytes) of a given JVM memory area.
# TYPE jvm_memory_bytes_max gauge
jvm_memory_bytes_max{area="heap",} 5.00170752E9
jvm_memory_bytes_max{area="nonheap",} -1.0
# HELP jvm_memory_pool_bytes_used Used bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_used gauge
jvm_memory_pool_bytes_used{pool="Code Cache",} 3.7606336E7
jvm_memory_pool_bytes_used{pool="Metaspace",} 6.10214E7
jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 7562680.0
jvm_memory_pool_bytes_used{pool="PS Eden Space",} 8.73788184E8
jvm_memory_pool_bytes_used{pool="PS Survivor Space",} 2.30681872E8
jvm_memory_pool_bytes_used{pool="PS Old Gen",} 4.14368384E8
# HELP jvm_memory_pool_bytes_committed Committed bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_committed gauge
jvm_memory_pool_bytes_committed{pool="Code Cache",} 3.7945344E7
jvm_memory_pool_bytes_committed{pool="Metaspace",} 6.4970752E7
jvm_memory_pool_bytes_committed{pool="Compressed Class Space",} 8871936.0
jvm_memory_pool_bytes_committed{pool="PS Eden Space",} 1.052246016E9
jvm_memory_pool_bytes_committed{pool="PS Survivor Space",} 3.70147328E8
jvm_memory_pool_bytes_committed{pool="PS Old Gen",} 3.579314176E9
# HELP jvm_memory_pool_bytes_max Max bytes of a given JVM memory pool.
# TYPE jvm_memory_pool_bytes_max gauge
jvm_memory_pool_bytes_max{pool="Code Cache",} 2.5165824E8
jvm_memory_pool_bytes_max{pool="Metaspace",} -1.0
jvm_memory_pool_bytes_max{pool="Compressed Class Space",} 1.073741824E9
jvm_memory_pool_bytes_max{pool="PS Eden Space",} 1.052246016E9
jvm_memory_pool_bytes_max{pool="PS Survivor Space",} 3.70147328E8
jvm_memory_pool_bytes_max{pool="PS Old Gen",} 3.579314176E9

@Gehel I still have to figure out what's wrong with jmx_exporter above, but what do you think re: metrics in P6392 ?

In T181627#3803312, @fgiunchedi wrote:

I tried jmx_exporter on deployment-logstash2 with the results below. A few notes: the exporter config needs to be somewhere accessible by elasticsearch (e.g. /srv/elasticsearch) or asking for metrics fails with something like [...]

/srv/elasticsearch/jmx_exporter.yaml looks more like a config file, I would expect to find it in /etc/elasticsearch.

In T181627#3827971, @fgiunchedi wrote:

@Gehel I still have to figure out what's wrong with jmx_exporter above, but what do you think re: metrics in P6392 ?

I did not do a full comparision with what we have in Graphite atm, but it looks like the most important metrics are there. We should probably keep both while we migrate the dashboards and see if we are missing something important.

A few notes about P6392:

• we seem to have some duplication with system metrics (like the free space on /srv/)
• we have some duplication with jmx_exporter (as in T181627#3803312)
  • GC is reported both in elasticsearch_exporter and in jmx_exporter. I much prefer to have GC metrics standardized across application (so exported through JMX, not through the elasticsearch specific exporter) (side note, the naming seems weird: elasticsearch_jvm_gc_collection_seconds_count is it a count of the number of GC, or a duration in seconds?)
• there seems to be some metrics about go garbage collection. As far as I know, we don't have anything running on go for the elasticsearch statck (or is it the elasticsaerch_exporter itself? it that's the case, do we really need those metrics?)
• we have both node level metrics (example: all the thread pools) and cluster level metrics (example: all the *cluster_health*). We probably don't want to collect cluster level metrics from all nodes, but to collect them from the LVS endpoint.

Mentioned in SAL (#wikimedia-operations) [2017-12-13T09:41:57Z] <godog> upload prometheus-elasticsearch-exporter to jessie-wikimedia - T181627

In T181627#3828220, @Gehel wrote:

In T181627#3803312, @fgiunchedi wrote:

I tried jmx_exporter on deployment-logstash2 with the results below. A few notes: the exporter config needs to be somewhere accessible by elasticsearch (e.g. /srv/elasticsearch) or asking for metrics fails with something like [...]

/srv/elasticsearch/jmx_exporter.yaml looks more like a config file, I would expect to find it in /etc/elasticsearch.

Indeed, that's what I tried first but didn't work. I'm not 100% sure why but I bet it has to do with jvm restrictions set up to avoid navigating the filesystem at will.

In T181627#3827971, @fgiunchedi wrote:

@Gehel I still have to figure out what's wrong with jmx_exporter above, but what do you think re: metrics in P6392 ?

I did not do a full comparision with what we have in Graphite atm, but it looks like the most important metrics are there. We should probably keep both while we migrate the dashboards and see if we are missing something important.

+1

A few notes about P6392:

• we seem to have some duplication with system metrics (like the free space on /srv/)
• we have some duplication with jmx_exporter (as in T181627#3803312)
  • GC is reported both in elasticsearch_exporter and in jmx_exporter. I much prefer to have GC metrics standardized across application (so exported through JMX, not through the elasticsearch specific exporter) (side note, the naming seems weird: elasticsearch_jvm_gc_collection_seconds_count is it a count of the number of GC, or a duration in seconds?)
• there seems to be some metrics about go garbage collection. As far as I know, we don't have anything running on go for the elasticsearch statck (or is it the elasticsaerch_exporter itself? it that's the case, do we really need those metrics?)
• we have both node level metrics (example: all the thread pools) and cluster level metrics (example: all the *cluster_health*). We probably don't want to collect cluster level metrics from all nodes, but to collect them from the LVS endpoint.

After a chat with @Gehel to shed some light we'll proceed with the following:

1. Start with deploying elasticsearch-exporter in each node (logstash first), collecting cluster-level metrics as well from all nodes. Get feedback for alerting and dashboards. Alerting will be adjusted/tested accordingly (e.g. for unallocated shards take the maximum, to avoid a shower of alerts from all hosts.
2. Deploy elasticsearch-exporter to cirrus es cluster as well.
3. [Likely next year] Figure out how to deploy jmx_exporter as well (more complicated, requires cluster restart)
   • Investigate the jmx_exporter config path issue (can't be under /etc/elasticsearch/ ?)
   • Investigate the fact that metrics are being reported by jmx_exporter but an error during scrape is reported jmx_scrape_error 1.0

Change 398025 had a related patch set uploaded (by Gehel; owner: Guillaume Lederrey):
[operations/puppet@production] elasticsearch: deploy prometheus-elasticsearch-exporter

https://gerrit.wikimedia.org/r/398025

Change 398026 had a related patch set uploaded (by Gehel; owner: Guillaume Lederrey):
[operations/puppet@production] logstash: activate prometheus elasticsearch exporter

https://gerrit.wikimedia.org/r/398026

Change 398027 had a related patch set uploaded (by Gehel; owner: Guillaume Lederrey):
[operations/puppet@production] elasticsearch: activate prometheus elasticsearch exporter

https://gerrit.wikimedia.org/r/398027

Change 398051 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: configure prometheus to collect metrics from elasticsearch

https://gerrit.wikimedia.org/r/398051

Change 398025 merged by Gehel:
[operations/puppet@production] elasticsearch: deploy prometheus-elasticsearch-exporter

https://gerrit.wikimedia.org/r/398025

Change 398026 merged by Gehel:
[operations/puppet@production] logstash: activate prometheus elasticsearch exporter

https://gerrit.wikimedia.org/r/398026

Change 398027 merged by Gehel:
[operations/puppet@production] elasticsearch: activate prometheus elasticsearch exporter

https://gerrit.wikimedia.org/r/398027

Change 398059 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch: configure prometheus to collect metrics from both logstash and elasticsearch

https://gerrit.wikimedia.org/r/398059

Change 398051 merged by Gehel:
[operations/puppet@production] elasticsearch: configure prometheus to collect metrics from logstash

https://gerrit.wikimedia.org/r/398051

Change 398059 merged by Gehel:
[operations/puppet@production] elasticsearch: configure prometheus to collect metrics from elasticsearch

https://gerrit.wikimedia.org/r/398059

elasticsearch_exporter is deployed on all elasticsearch nodes. Still to do:

• same work on jmx_exporter
• update grafana dashboards

Now that I understand a bit better how prometheus works, the jmx_exporter starts to be scary. If I understand correctly, it exposes all MBeans and filtering is done after the fact. We should really never use the jmx_exporter without a whitelist.

Querying all MBeans can be dangerous. MBean can expose anything, including stuff that is expensive to compute, blocking, or even potentially crash the application. JMX is (too) powerful...

While updating the Shards graph, I found that the way shards are exposed through prometheus is not optimal. We have different metric names for different shard states:

• elasticsearch_cluster_health_active_shards
• elasticsearch_cluster_health_delayed_unassigned_shards
• elasticsearch_cluster_health_initializing_shards
• ...

It seems that it would be simpler to query them if we had a metric named elasticsearch_cluster_health_shards with labels for the different states (active, delayed_unassigned, initializing, ...).

The way the metrics are exposed seems to be linked to way the elasticsearch_exporter exposes them, but there might be an easy way to rename those. @fgiunchedi your input would be welcomed.

In T181627#3836838, @Gehel wrote:

While updating the Shards graph, I found that the way shards are exposed through prometheus is not optimal. We have different metric names for different shard states:

• elasticsearch_cluster_health_active_shards
• elasticsearch_cluster_health_delayed_unassigned_shards
• elasticsearch_cluster_health_initializing_shards
• ...

It seems that it would be simpler to query them if we had a metric named elasticsearch_cluster_health_shards with labels for the different states (active, delayed_unassigned, initializing, ...).

The way the metrics are exposed seems to be linked to way the elasticsearch_exporter exposes them, but there might be an easy way to rename those. @fgiunchedi your input would be welcomed.

I'm not sure it applies here but the general recommendation for having dimensions in labels as opposed to metric names is whether sum() on the metric would yield a meaningful number (e.g. no double counting). Does the split make more sense with that in mind?
WRT dashboarding and working with metrics like the above, you can also use __name__ as a label to match the metric name, for example {__name__=~'elasticsearch_cluster_health_(active|initializing)_shards'}

The sum of all shard states does make some kind of sense: the some of all states should give the total number of shards, but the total is also exposed... so not sure.

The __name__ trick should work in my case. So let's leave it at that atm. Thanks!

Looking at the prometheus-jmx-exporter .deb, it seems to depend on default-jre which is openjdk-7-jre on Jessie. We use OpenJDK 8 for elasticsearch (and wdqs, and hopefully for most of our Java based stack). Having 2 different JRE installed is a source of unnecessary pain. My understanding of Debian packaging is low enough that I don't really know how this should be fixed. Maybe depend on java7-runtime instead of default-jre since both openjdk-7 and openjdk-8 provide it? Or remove the JRE dependency completely, since the application being monitored is responsible to depend on the appropriate JRE (this is true for the agent part of the exporter, not for the HTTP server part).

I noticed the same re: jre dependencies and fixed it in https://gerrit.wikimedia.org/r/#/c/394322/ though that version isn't built/uploaded yet. @Ottomata what's the procedure you are using for those? I see versions with ~jessie1 and ~stretch1 uploaded ATM

While migrating existing grafana dashboards, it looks like some dashboards are broken and most probably unused. We should delete them instead of taking time to migrate:

• elasticsearch load testing
• elasticsearch percentiles beta
• elasticsearch cluster recovery investigation

@EBernhardson do you want to keep one of those?

load testing and percentiles could certainly go away. cluster recovery might be useful at some point in the future but hard to say. All of that data is found in other dashboards though, just not broken out by server and in a single board. It certainly wont be immediately useful, and if the data is there it can be recreated as necessary.

Unrelated to dashboards, but for prometheus. We will likely need a fork (or an additional custom collector) to collect extra metrics that are only reported by our cluster. Specifically we collect per-node latency percentiles into a custom api endpoint on the elasticsearch servers. This isn't even in diamond yet as we only recently upgraded the plugin version on the cluster to expose these metrics.

To build:

https://github.com/wikimedia/operations-debs-prometheus-jmx-exporter/blob/master/debian/README.Debian

I'm probably doing somethign wrong with the ~jessie1 and ~stretch1 versions. It's JVM right? So the same build should be good for both distros? Not sure. I've just always done that and didn't bother to check if I shouldn't.

Hm, looks like the last line of README.debian got cut off. Should be:

USENETWORK=yes GIT_PBUILDER_AUTOCONF=no DIST=stretch WIKIMEDIA=yes gbp buildpackage -sa -us -uc --git-builder=git-pbuilder

I ported elasticsearch-memory and elasticsearch-indexing.

• https://grafana-admin.wikimedia.org/dashboard/db/elasticsearch-memory-prometheus
• https://grafana-admin.wikimedia.org/dashboard/db/elasticsearch-indexing-prometheus

Some metrics where unavailable (not speaking about mediawiki counters) with prometheus so I flagged thes graph as (prometheus todo).

available in graphite but missing in prometheus (elastic specific):

• elasticsearch.indices.segments.terms_memory_in_bytes
• elasticsearch.indices.segments.index_writer_memory_in_bytes
• elasticsearch.indices.completion.size_in_bytes
• elasticsearch.indices.segments.norms_memory_in_bytes
• elasticsearch.indices.segments.stored_fields_memory_in_bytes
• elasticsearch.indices.segments.doc_values_memory_in_bytes
• elasticsearch.indices.segments.fixed_bit_set_memory_in_bytes

iostats:

• iostat.{device}.iops, used node_disk_reads_completed+node_disk_writes_completed instead
• iostat.average_queue_length, used node_disk_io_now but this one is not very precise and the curve is always flat

Additional missing metrics:

• elasticsearch.indices.search.groups.prefix.query_total
• elasticsearch.indices.search.groups.prefix.query_time_in_millis
• elasticsearch.indices.suggest.total
• elasticsearch.indices.suggest.time_in_millis
• elasticsearch.indices.search.groups.full_text.query_total
• elasticsearch.indices.search.groups.full_text.query_time_in_millis
• elasticsearch.indices.search.groups.more_like.query_total
• elasticsearch.indices.search.groups.more_like.query_time_in_millis

Issue opened upstream to include those metrics: https://github.com/justwatchcom/elasticsearch_exporter/issues/115

In T181627#3845066, @Ottomata wrote:

I'm probably doing somethign wrong with the ~jessie1 and ~stretch1 versions. It's JVM right? So the same build should be good for both distros? Not sure. I've just always done that and didn't bother to check if I shouldn't.

While elasticsearch is JVM, the exporter is Go, runs outside the JVM and uses the elasticsearch HTTP API to collect metrics. As I understand it, Go statically links everything, so the same build should still be good for both distro (but please check me on that).

As I understand it, Go statically links everything, so the same build should still be good for both distro (but please check me on that).

That's correct. For jessie we couldn't even build these since the toolchain is fairly immature there.

While elasticsearch is JVM, the exporter is Go ?

Huh, I thought we were talking about prometheus-jmx-exporter, which is a java project: https://github.com/wikimedia/operations-debs-prometheus-jmx-exporter/tree/master/jmx_prometheus_httpserver/src/main/java/io/prometheus/jmx

Unfortunately all of the elasticsearch-specific metrics are no exposed over jmx. We can get generic JVM info that way, but for the specialized stats we have to query the elasticsearch APIs.

Ah ok, my response was to Filippo asking how to build prometheus-jmx-exporter.

Short summary of the status of this task (since quite a lot of discussion happened):

• the prometheus elasticsearch_exporter is deployed everywhere
• some metrics are missing, issue is opened upstream (https://github.com/justwatchcom/elasticsearch_exporter/issues/115)
• some metrics are specific to us (per node latency percentiles), we need to implement a specific collector for those (this should be done in another task: T183451)
• copies of all existing dashboards have been created, using the Prometheus metrics
• the old dashboards should be deleted once we have enough historical data (1 month?)
• jmx_exporter has not yet been deployed or used in any way

A .deb of prometheus jxm_exporter is now available. I started to experiment on deployment-elastic06. Elasticsearch installs a fairly strict security manager that prevents the jmx_exporter agent from working properly. Adding a /home/elasticsearch/.java.policy file with the content below solves the issue:

grant codeBase "file:/usr/share/java/prometheus/-" {
  permission javax.management.MBeanServerPermission "createMBeanServer";
  permission javax.management.MBeanPermission "*", "*";
  permission java.lang.RuntimePermission "accessClassInPackage.sun.management";
  permission java.io.FilePermission "/etc/elasticsearch/prometheus_jmx_exporter.yaml", "read";
  permission java.io.FilePermission "/proc/self/status", "read";
};

It isn't very clear from the jmx_exporter documentation, but in addition to the MBeans that you configure, the exporter always exposes some standard JVM metrics. Those seem sufficient as JVM level metrics, and provide a good naming and help messages.

Change 402095 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] elasticsearch / prometheus: enable prometheus jmx_exporter

https://gerrit.wikimedia.org/r/402095

Playing with jmx_exporter and elasticsearch, it looks like the metrics exposed through the elasticsearch API are already sufficient, and the naming is consistent between jmx_exporter and elasticsearch_exporter. So we can probably drop the jmx_exporter for elasticsearch.

Upstream has released version 1.0.2 with the additional elasticsearch metrics that we need: https://github.com/justwatchcom/elasticsearch_exporter/tree/v1.0.2

elasticsearch exporter has been upgraded across all elasticsearch nodes. We should now have all the metrics we need. We can close this task and reopen if we find something we miss.

Change 402095 abandoned by Gehel:
elasticsearch / prometheus: enable prometheus jmx_exporter

Reason:
Correct, we have all the metrics we need through the elasticsearch_exporter. Dropping this change.

https://gerrit.wikimedia.org/r/402095