Page MenuHomePhabricator

The metrics of the streaming-updater-consumer should be visible in grafana
Closed, ResolvedPublic

Description

As a maintainer of the wdqs streaming-updater I want to see the metrics it produces in grafana so that I can create dashboards to visualize how it behaves.

On wdqs1009 where the consumer is enable the metrics are properly generated and exposed through a prometheus exporter on port 9101:

dcausse@wdqs1009:~$ curl -s localhost:9101 | grep ^wdqs_str
wdqs_streaming_updater_rdf_store_time_cnt_Count 2.4398143E7
wdqs_streaming_updater_kafka_stream_consumer_triples_accumulated_Count 1.0062722E7
wdqs_streaming_updater_divergences_Count 44242.0
wdqs_streaming_updater_mutations_Count 8981755.0
wdqs_streaming_updater_poll_time_cnt_Count 1.11013286E8
wdqs_streaming_updater_shared_element_mutations_Count 278521.0
wdqs_streaming_updater_delete_mutations_Count 27054.0
wdqs_streaming_updater_shared_element_redundant_mutations_Count 276446.0
wdqs_streaming_updater_kafka_stream_consumer_triples_offered_Count 9767872.0
wdqs_streaming_kafka_stream_consumer_lag_Value 99265.0

But these metrics are not visible in grafana and perhaps not even collected by prometheus.

AC:

  • metrics produced by the prometheus exporter (port 9101) should be accessible in prometheus and grafana

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
CBogen triaged this task as High priority.Nov 2 2020, 3:12 PM
CBogen moved this task from Incoming to Scaling on the Wikidata-Query-Service board.

Change 643941 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/puppet@production] [wdqs] proper selector for machines running the streaming-updater

https://gerrit.wikimedia.org/r/643941

Change 643941 merged by Ryan Kemper:
[operations/puppet@production] [wdqs] proper selector for machines running the streaming-updater

https://gerrit.wikimedia.org/r/643941

Change 646621 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/puppet@production] [wdqs] proper selector for machines running the streaming-updater (take 2)

https://gerrit.wikimedia.org/r/646621

Change 646621 merged by Ryan Kemper:
[operations/puppet@production] [wdqs] proper selector for machines running the streaming-updater (take 2)

https://gerrit.wikimedia.org/r/646621

Change 649827 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/puppet@production] [wdqs] add jmx_wdqs_streaming_updater prometheus job

https://gerrit.wikimedia.org/r/649827

Change 649827 merged by Ryan Kemper:
[operations/puppet@production] [wdqs] add jmx_wdqs_streaming_updater prometheus job

https://gerrit.wikimedia.org/r/649827

Gehel claimed this task.