To be deployed on DSE wdqs-proxy must hint k8s to scrape prometheus metrics
We should should report general service health as well as specific metrics defined by us (histogram of latencies, counters status, etc).
Description
Description
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | gmodena | T422522 WE2.5.8 WDQS v2 technical build | |||
| Resolved | gmodena | T424844 wdqs-proxy should integrate with prometheus |
Event Timeline
Comment Actions
This is an example of how node services export metrics https://gitlab.wikimedia.org/repos/data-engineering/service-utils#metrics
Comment Actions
Integrating quarkus-micrometer-registry-prometheus is straightforward and exposes a bunch of metrics out of the box
Default metrics
Note: we should prefix and/or tag these metrics as wdqs. We also might not need all of this right away.
% curl http://localhost:9100/metrics
# TYPE worker_pool_idle gauge
# HELP worker_pool_idle The number of resources from the pool currently used
worker_pool_idle{pool_name="vert.x-internal-blocking",pool_type="worker"} 20.0
worker_pool_idle{pool_name="vert.x-worker-thread",pool_type="worker"} 199.0
# TYPE process_uptime_seconds gauge
# HELP process_uptime_seconds The uptime of the Java virtual machine
process_uptime_seconds 6.151
# TYPE worker_pool_rejected counter
# HELP worker_pool_rejected Number of times submissions to the pool have been rejected
worker_pool_rejected_total{pool_name="vert.x-internal-blocking",pool_type="worker"} 0.0
worker_pool_rejected_total{pool_name="vert.x-worker-thread",pool_type="worker"} 0.0
# TYPE jvm_memory_used_bytes gauge
# HELP jvm_memory_used_bytes The amount of used memory
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'profiled nmethods'"} 3791488.0
jvm_memory_used_bytes{area="heap",id="G1 Survivor Space"} 8719920.0
jvm_memory_used_bytes{area="heap",id="G1 Old Gen"} 1183744.0
jvm_memory_used_bytes{area="nonheap",id="Metaspace"} 2.4553456E7
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-nmethods'"} 2173184.0
jvm_memory_used_bytes{area="heap",id="G1 Eden Space"} 4.194304E7
jvm_memory_used_bytes{area="nonheap",id="Compressed Class Space"} 3243232.0
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'"} 741632.0
# TYPE http_server_active_connections gauge
# HELP http_server_active_connections Number of opened connections to the HTTP Server
http_server_active_connections{server_port="8090"} 0.0
http_server_active_connections{server_port="9100"} 1.0
# TYPE worker_pool_ratio gauge
# HELP worker_pool_ratio Pool usage ratio
worker_pool_ratio{pool_name="vert.x-internal-blocking",pool_type="worker"} NaN
worker_pool_ratio{pool_name="vert.x-worker-thread",pool_type="worker"} 0.005
# TYPE jvm_memory_usage_after_gc gauge
# HELP jvm_memory_usage_after_gc The percentage of long-lived heap pool used after the last GC event, in the range [0..1]
jvm_memory_usage_after_gc{area="heap",pool="long-lived"} 7.059193659329665E-5
# TYPE jvm_info counter
# HELP jvm_info JVM version info
jvm_info_total{runtime="OpenJDK Runtime Environment",vendor="Oracle Corporation",version="25.0.2+10"} 1.0
# TYPE netty_allocator_pooled_chunk_size gauge
# HELP netty_allocator_pooled_chunk_size
netty_allocator_pooled_chunk_size{allocator_type="PooledByteBufAllocator",name="vertx-pooled"} 65536.0
netty_allocator_pooled_chunk_size{allocator_type="PooledByteBufAllocator",name="pooled"} 65536.0
# TYPE jvm_memory_max_bytes gauge
# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
jvm_memory_max_bytes{area="nonheap",id="CodeHeap 'profiled nmethods'"} 1.22023936E8
jvm_memory_max_bytes{area="heap",id="G1 Survivor Space"} -1.0
jvm_memory_max_bytes{area="heap",id="G1 Old Gen"} 1.6768827392E10
jvm_memory_max_bytes{area="nonheap",id="Metaspace"} -1.0
jvm_memory_max_bytes{area="nonheap",id="CodeHeap 'non-nmethods'"} 7602176.0
jvm_memory_max_bytes{area="heap",id="G1 Eden Space"} -1.0
jvm_memory_max_bytes{area="nonheap",id="Compressed Class Space"} 1.073741824E9
jvm_memory_max_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'"} 1.22032128E8
# TYPE jvm_buffer_total_capacity_bytes gauge
# HELP jvm_buffer_total_capacity_bytes An estimate of the total capacity of the buffers in this pool
jvm_buffer_total_capacity_bytes{id="mapped - 'non-volatile memory'"} 0.0
jvm_buffer_total_capacity_bytes{id="mapped"} 0.0
jvm_buffer_total_capacity_bytes{id="direct"} 65567.0
# TYPE jvm_classes_loaded_count_classes counter
# HELP jvm_classes_loaded_count_classes The number of classes loaded in the Java virtual machine
jvm_classes_loaded_count_classes_total 5793.0
# TYPE jvm_gc_live_data_size_bytes gauge
# HELP jvm_gc_live_data_size_bytes Size of long-lived heap memory pool after reclamation
jvm_gc_live_data_size_bytes 0.0
# TYPE http_server_active_requests gauge
# HELP http_server_active_requests
http_server_active_requests{server_port="9100",url_scheme="http"} 1.0
http_server_active_requests{server_port="8090",url_scheme="http"} 0.0
# TYPE netty_allocator_pooled_arenas gauge
# HELP netty_allocator_pooled_arenas
netty_allocator_pooled_arenas{allocator_type="PooledByteBufAllocator",memory_type="heap",name="pooled"} 40.0
netty_allocator_pooled_arenas{allocator_type="PooledByteBufAllocator",memory_type="heap",name="vertx-pooled"} 40.0
netty_allocator_pooled_arenas{allocator_type="PooledByteBufAllocator",memory_type="direct",name="pooled"} 40.0
netty_allocator_pooled_arenas{allocator_type="PooledByteBufAllocator",memory_type="direct",name="vertx-pooled"} 40.0
# TYPE system_load_average_1m gauge
# HELP system_load_average_1m The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
system_load_average_1m 1.875
# TYPE jvm_buffer_memory_used_bytes gauge
# HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
jvm_buffer_memory_used_bytes{id="mapped - 'non-volatile memory'"} 0.0
jvm_buffer_memory_used_bytes{id="mapped"} 0.0
jvm_buffer_memory_used_bytes{id="direct"} 65568.0
# TYPE process_cpu_time_ns counter
# HELP process_cpu_time_ns The \"cpu time\" used by the Java Virtual Machine process
process_cpu_time_ns_total 4.51E9
# TYPE jvm_threads_states_threads gauge
# HELP jvm_threads_states_threads The current number of threads
jvm_threads_states_threads{state="runnable"} 25.0
jvm_threads_states_threads{state="blocked"} 0.0
jvm_threads_states_threads{state="waiting"} 3.0
jvm_threads_states_threads{state="timed-waiting"} 2.0
jvm_threads_states_threads{state="new"} 0.0
jvm_threads_states_threads{state="terminated"} 0.0
# TYPE jvm_gc_concurrent_phase_time_seconds summary
# HELP jvm_gc_concurrent_phase_time_seconds Time spent in concurrent phase
jvm_gc_concurrent_phase_time_seconds_count{action="end of concurrent GC pause",cause="No GC",gc="G1 Concurrent GC"} 2.0
jvm_gc_concurrent_phase_time_seconds_sum{action="end of concurrent GC pause",cause="No GC",gc="G1 Concurrent GC"} 0.005
# TYPE jvm_gc_concurrent_phase_time_seconds_max gauge
# HELP jvm_gc_concurrent_phase_time_seconds_max Time spent in concurrent phase
jvm_gc_concurrent_phase_time_seconds_max{action="end of concurrent GC pause",cause="No GC",gc="G1 Concurrent GC"} 0.005
# TYPE jvm_threads_started_threads counter
# HELP jvm_threads_started_threads The total number of application threads started in the JVM
jvm_threads_started_threads_total 30.0
# TYPE process_files_open_files gauge
# HELP process_files_open_files The open file descriptor count
process_files_open_files 76.0
# TYPE http_server_connections_seconds_max gauge
# HELP http_server_connections_seconds_max The duration of the connections
http_server_connections_seconds_max{server_port="8090"} 0.0
http_server_connections_seconds_max{server_port="9100"} 0.058028329
# TYPE http_server_connections_seconds summary
# HELP http_server_connections_seconds The duration of the connections
http_server_connections_seconds_active_count{server_port="8090"} 0.0
http_server_connections_seconds_duration_sum{server_port="8090"} 0.0
http_server_connections_seconds_active_count{server_port="9100"} 1.0
http_server_connections_seconds_duration_sum{server_port="9100"} 0.057983309
# TYPE jvm_memory_committed_bytes gauge
# HELP jvm_memory_committed_bytes The amount of memory in bytes that is committed for the Java virtual machine to use
jvm_memory_committed_bytes{area="nonheap",id="CodeHeap 'profiled nmethods'"} 3866624.0
jvm_memory_committed_bytes{area="heap",id="G1 Survivor Space"} 1.6777216E7
jvm_memory_committed_bytes{area="heap",id="G1 Old Gen"} 8388608.0
jvm_memory_committed_bytes{area="nonheap",id="Metaspace"} 2.5034752E7
jvm_memory_committed_bytes{area="nonheap",id="CodeHeap 'non-nmethods'"} 3473408.0
jvm_memory_committed_bytes{area="heap",id="G1 Eden Space"} 5.8720256E7
jvm_memory_committed_bytes{area="nonheap",id="Compressed Class Space"} 3407872.0
jvm_memory_committed_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'"} 2555904.0
# TYPE jvm_classes_unloaded_classes counter
# HELP jvm_classes_unloaded_classes The number of classes unloaded in the Java virtual machine
jvm_classes_unloaded_classes_total 0.0
# TYPE process_files_max_files gauge
# HELP process_files_max_files The maximum file descriptor count
process_files_max_files 524288.0
# TYPE jvm_gc_overhead gauge
# HELP jvm_gc_overhead An approximation of the percent of CPU time used by GC activities over the last lookback period or since monitoring began, whichever is shorter, in the range [0..1]
jvm_gc_overhead 0.0031716471730075385
# TYPE netty_allocator_pooled_cache_size gauge
# HELP netty_allocator_pooled_cache_size
netty_allocator_pooled_cache_size{allocator_type="PooledByteBufAllocator",cache_type="small",name="pooled"} 256.0
netty_allocator_pooled_cache_size{allocator_type="PooledByteBufAllocator",cache_type="normal",name="vertx-pooled"} 64.0
netty_allocator_pooled_cache_size{allocator_type="PooledByteBufAllocator",cache_type="normal",name="pooled"} 64.0
netty_allocator_pooled_cache_size{allocator_type="PooledByteBufAllocator",cache_type="small",name="vertx-pooled"} 256.0
# TYPE worker_pool_queue_delay_seconds_max gauge
# HELP worker_pool_queue_delay_seconds_max Time spent in the waiting queue before being processed
worker_pool_queue_delay_seconds_max{pool_name="vert.x-internal-blocking",pool_type="worker"} 0.0
worker_pool_queue_delay_seconds_max{pool_name="vert.x-worker-thread",pool_type="worker"} 0.001716622
# TYPE worker_pool_queue_delay_seconds summary
# HELP worker_pool_queue_delay_seconds Time spent in the waiting queue before being processed
worker_pool_queue_delay_seconds_count{pool_name="vert.x-internal-blocking",pool_type="worker"} 0.0
worker_pool_queue_delay_seconds_sum{pool_name="vert.x-internal-blocking",pool_type="worker"} 0.0
worker_pool_queue_delay_seconds_count{pool_name="vert.x-worker-thread",pool_type="worker"} 1.0
worker_pool_queue_delay_seconds_sum{pool_name="vert.x-worker-thread",pool_type="worker"} 0.001716622
# TYPE worker_pool_completed counter
# HELP worker_pool_completed Number of times resources from the pool have been acquired
worker_pool_completed_total{pool_name="vert.x-internal-blocking",pool_type="worker"} 0.0
worker_pool_completed_total{pool_name="vert.x-worker-thread",pool_type="worker"} 0.0
# TYPE netty_allocator_memory_used gauge
# HELP netty_allocator_memory_used
netty_allocator_memory_used{allocator_type="UnpooledByteBufAllocator",memory_type="direct",name="quarkus-multipart-form-upload"} 0.0
netty_allocator_memory_used{allocator_type="UnpooledByteBufAllocator",memory_type="direct",name="vertx-unpooled"} 0.0
netty_allocator_memory_used{allocator_type="PooledByteBufAllocator",memory_type="heap",name="pooled"} 0.0
netty_allocator_memory_used{allocator_type="UnpooledByteBufAllocator",memory_type="heap",name="quarkus-multipart-form-upload"} 0.0
netty_allocator_memory_used{allocator_type="UnpooledByteBufAllocator",memory_type="direct",name="unpooled"} 31.0
netty_allocator_memory_used{allocator_type="PooledByteBufAllocator",memory_type="heap",name="vertx-pooled"} 0.0
netty_allocator_memory_used{allocator_type="PooledByteBufAllocator",memory_type="direct",name="pooled"} 65536.0
netty_allocator_memory_used{allocator_type="UnpooledByteBufAllocator",memory_type="heap",name="unpooled"} 128.0
netty_allocator_memory_used{allocator_type="UnpooledByteBufAllocator",memory_type="heap",name="vertx-unpooled"} 0.0
netty_allocator_memory_used{allocator_type="PooledByteBufAllocator",memory_type="direct",name="vertx-pooled"} 0.0
# TYPE netty_eventexecutor_workers gauge
# HELP netty_eventexecutor_workers Number of event executor workers
netty_eventexecutor_workers 1.0
# TYPE jvm_gc_pause_seconds summary
# HELP jvm_gc_pause_seconds Time spent in GC pause
jvm_gc_pause_seconds_count{action="end of minor GC",cause="Metadata GC Threshold",gc="G1 Young Generation"} 1.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="Metadata GC Threshold",gc="G1 Young Generation"} 0.007
jvm_gc_pause_seconds_count{action="end of minor GC",cause="G1 Evacuation Pause",gc="G1 Young Generation"} 1.0
jvm_gc_pause_seconds_sum{action="end of minor GC",cause="G1 Evacuation Pause",gc="G1 Young Generation"} 0.009
# TYPE jvm_gc_pause_seconds_max gauge
# HELP jvm_gc_pause_seconds_max Time spent in GC pause
jvm_gc_pause_seconds_max{action="end of minor GC",cause="Metadata GC Threshold",gc="G1 Young Generation"} 0.007
jvm_gc_pause_seconds_max{action="end of minor GC",cause="G1 Evacuation Pause",gc="G1 Young Generation"} 0.009
# TYPE process_start_time_seconds gauge
# HELP process_start_time_seconds Start time of the process since unix epoch.
process_start_time_seconds 1.777929829292E9
# TYPE worker_pool_usage_seconds summary
# HELP worker_pool_usage_seconds Time spent using resources from the pool
worker_pool_usage_seconds_count{pool_name="vert.x-internal-blocking",pool_type="worker"} 0.0
worker_pool_usage_seconds_sum{pool_name="vert.x-internal-blocking",pool_type="worker"} 0.0
worker_pool_usage_seconds_count{pool_name="vert.x-worker-thread",pool_type="worker"} 0.0
worker_pool_usage_seconds_sum{pool_name="vert.x-worker-thread",pool_type="worker"} 0.0
# TYPE worker_pool_usage_seconds_max gauge
# HELP worker_pool_usage_seconds_max Time spent using resources from the pool
worker_pool_usage_seconds_max{pool_name="vert.x-internal-blocking",pool_type="worker"} 0.0
worker_pool_usage_seconds_max{pool_name="vert.x-worker-thread",pool_type="worker"} 0.0
# TYPE worker_pool_queue_size gauge
# HELP worker_pool_queue_size Number of pending elements in the waiting queue
worker_pool_queue_size{pool_name="vert.x-internal-blocking",pool_type="worker"} 0.0
worker_pool_queue_size{pool_name="vert.x-worker-thread",pool_type="worker"} 0.0
# TYPE jvm_gc_memory_allocated_bytes counter
# HELP jvm_gc_memory_allocated_bytes Incremented for an increase in the size of the (young) heap memory pool after one GC to before the next
jvm_gc_memory_allocated_bytes_total 1.00663296E8
# TYPE jvm_classes_loaded_classes gauge
# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
jvm_classes_loaded_classes 5822.0
# TYPE process_cpu_usage gauge
# HELP process_cpu_usage The \"recent cpu usage\" for the Java Virtual Machine process
process_cpu_usage 0.0
# TYPE worker_pool_active gauge
# HELP worker_pool_active The number of resources from the pool currently used
worker_pool_active{pool_name="vert.x-internal-blocking",pool_type="worker"} 0.0
worker_pool_active{pool_name="vert.x-worker-thread",pool_type="worker"} 1.0
# TYPE system_cpu_count gauge
# HELP system_cpu_count The number of processors available to the Java virtual machine
system_cpu_count 20.0
# TYPE netty_eventexecutor_tasks_pending gauge
# HELP netty_eventexecutor_tasks_pending
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-18"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-19"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-10"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-9"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-11"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-8"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-12"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-7"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-13"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-6"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-14"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-5"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-15"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-4"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-16"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-3"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-17"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-2"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-1"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-eventloop-thread-0"} 0.0
netty_eventexecutor_tasks_pending{name="vert.x-acceptor-thread-0"} 0.0
# TYPE jvm_threads_live_threads gauge
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
jvm_threads_live_threads 30.0
# TYPE http_server_bytes_read summary
# HELP http_server_bytes_read Number of bytes received by the server
http_server_bytes_read_count{server_port="8090"} 0.0
http_server_bytes_read_sum{server_port="8090"} 0.0
http_server_bytes_read_count{server_port="9100"} 0.0
http_server_bytes_read_sum{server_port="9100"} 0.0
# TYPE http_server_bytes_read_max gauge
# HELP http_server_bytes_read_max Number of bytes received by the server
http_server_bytes_read_max{server_port="8090"} 0.0
http_server_bytes_read_max{server_port="9100"} 0.0
# TYPE jvm_buffer_count_buffers gauge
# HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool
jvm_buffer_count_buffers{id="mapped - 'non-volatile memory'"} 0.0
jvm_buffer_count_buffers{id="mapped"} 0.0
jvm_buffer_count_buffers{id="direct"} 5.0
# TYPE jvm_gc_memory_promoted_bytes counter
# HELP jvm_gc_memory_promoted_bytes Count of positive increases in the size of the old generation memory pool before GC to after GC
jvm_gc_memory_promoted_bytes_total 0.0
# TYPE netty_allocator_memory_pinned gauge
# HELP netty_allocator_memory_pinned
netty_allocator_memory_pinned{allocator_type="PooledByteBufAllocator",memory_type="heap",name="pooled"} 0.0
netty_allocator_memory_pinned{allocator_type="PooledByteBufAllocator",memory_type="heap",name="vertx-pooled"} 0.0
netty_allocator_memory_pinned{allocator_type="PooledByteBufAllocator",memory_type="direct",name="pooled"} 0.0
netty_allocator_memory_pinned{allocator_type="PooledByteBufAllocator",memory_type="direct",name="vertx-pooled"} 0.0
# TYPE jvm_threads_peak_threads gauge
# HELP jvm_threads_peak_threads The peak live thread count since the Java virtual machine started or peak was reset
jvm_threads_peak_threads 30.0
# TYPE netty_allocator_pooled_threadlocal_caches gauge
# HELP netty_allocator_pooled_threadlocal_caches
netty_allocator_pooled_threadlocal_caches{allocator_type="PooledByteBufAllocator",name="vertx-pooled"} 0.0
netty_allocator_pooled_threadlocal_caches{allocator_type="PooledByteBufAllocator",name="pooled"} 1.0
# TYPE jvm_gc_max_data_size_bytes gauge
# HELP jvm_gc_max_data_size_bytes Max size of long-lived heap memory pool
jvm_gc_max_data_size_bytes 1.6768827392E10
# TYPE system_cpu_usage gauge
# HELP system_cpu_usage The \"recent cpu usage\" of the system the application is running in
system_cpu_usage 0.18181818181818182
# TYPE jvm_threads_daemon_threads gauge
# HELP jvm_threads_daemon_threads The current number of live daemon threads
jvm_threads_daemon_threads 8.0
# TYPE http_server_bytes_written_max gauge
# HELP http_server_bytes_written_max Number of bytes sent by the server
http_server_bytes_written_max{server_port="8090"} 0.0
http_server_bytes_written_max{server_port="9100"} 0.0
# TYPE http_server_bytes_written summary
# HELP http_server_bytes_written Number of bytes sent by the server
http_server_bytes_written_count{server_port="8090"} 0.0
http_server_bytes_written_sum{server_port="8090"} 0.0
http_server_bytes_written_count{server_port="9100"} 0.0
http_server_bytes_written_sum{server_port="9100"} 0.0
# EOFComment Actions
Custom metrics
Bike-shedding needed.
Injecting custom metrics is straightforward too. Broadly, I'm thinking of:
- Prefixing metrics with wdsq_<component>. For this service, wdqs_proxy
- Tagging query-level metrics with a type (e.g. SELECT, ASK), outcome (that represents success/client error/server error classes).
- Bucketing with latency ranges at single query type level as well as general. Note: the buckets in the example below need fine-tuning (we can extrapolate from query log stats). Maybe bucketing latency at query type is overkill and counts of success/client error/server error classes could be enough? I worry a bit about cardinality exploding.
- Reporting errors per operation/handler type. Maybe we should rename wdqs_proxy_rewrite_* to something more "general purpose" (wdqs_proxy_inspect_* ?).
# HELP wdqs_proxy_query_duration_milliseconds SPARQL query end-to-end latency
# TYPE wdqs_proxy_query_duration_milliseconds histogram
wdqs_proxy_query_duration_milliseconds_bucket{query_type="SELECT",outcome="success",le="1.0"} 0
wdqs_proxy_query_duration_milliseconds_bucket{query_type="SELECT",outcome="success",le="5.0"} 4
wdqs_proxy_query_duration_milliseconds_bucket{query_type="SELECT",outcome="success",le="10.0"} 18
wdqs_proxy_query_duration_milliseconds_bucket{query_type="SELECT",outcome="success",le="50.0"} 110
wdqs_proxy_query_duration_milliseconds_bucket{query_type="SELECT",outcome="success",le="100.0"} 198
wdqs_proxy_query_duration_milliseconds_bucket{query_type="SELECT",outcome="success",le="500.0"} 247
wdqs_proxy_query_duration_milliseconds_bucket{query_type="SELECT",outcome="success",le="1000.0"} 251
wdqs_proxy_query_duration_milliseconds_bucket{query_type="SELECT",outcome="success",le="+Inf"} 251
wdqs_proxy_query_duration_milliseconds_count{query_type="SELECT",outcome="success"} 251
wdqs_proxy_query_duration_milliseconds_sum{query_type="SELECT",outcome="success"} 18423.0
wdqs_proxy_query_duration_milliseconds_bucket{query_type="SELECT",outcome="bad_request",le="1.0"} 1
wdqs_proxy_query_duration_milliseconds_bucket{query_type="SELECT",outcome="bad_request",le="5.0"} 7
wdqs_proxy_query_duration_milliseconds_bucket{query_type="SELECT",outcome="bad_request",le="+Inf"} 9
wdqs_proxy_query_duration_milliseconds_count{query_type="SELECT",outcome="bad_request"} 9
wdqs_proxy_query_duration_milliseconds_sum{query_type="SELECT",outcome="bad_request"} 21.0
wdqs_proxy_query_duration_milliseconds_bucket{query_type="SELECT",outcome="server_error",le="5.0"} 0
wdqs_proxy_query_duration_milliseconds_bucket{query_type="SELECT",outcome="server_error",le="50.0"} 2
wdqs_proxy_query_duration_milliseconds_bucket{query_type="SELECT",outcome="server_error",le="+Inf"} 2
wdqs_proxy_query_duration_milliseconds_count{query_type="SELECT",outcome="server_error"} 2
wdqs_proxy_query_duration_milliseconds_sum{query_type="SELECT",outcome="server_error"} 67.0
wdqs_proxy_query_duration_milliseconds_bucket{query_type="ASK",outcome="success",le="1.0"} 2
wdqs_proxy_query_duration_milliseconds_bucket{query_type="ASK",outcome="success",le="5.0"} 14
wdqs_proxy_query_duration_milliseconds_bucket{query_type="ASK",outcome="success",le="10.0"} 29
wdqs_proxy_query_duration_milliseconds_bucket{query_type="ASK",outcome="success",le="50.0"} 41
wdqs_proxy_query_duration_milliseconds_bucket{query_type="ASK",outcome="success",le="+Inf"} 41
wdqs_proxy_query_duration_milliseconds_count{query_type="ASK",outcome="success"} 41
wdqs_proxy_query_duration_milliseconds_sum{query_type="ASK",outcome="success"} 812.0
wdqs_proxy_query_duration_milliseconds_bucket{query_type="ASK",outcome="bad_request",le="1.0"} 0
wdqs_proxy_query_duration_milliseconds_bucket{query_type="ASK",outcome="bad_request",le="+Inf"} 1
wdqs_proxy_query_duration_milliseconds_count{query_type="ASK",outcome="bad_request"} 1
wdqs_proxy_query_duration_milliseconds_sum{query_type="ASK",outcome="bad_request"} 3.0
wdqs_proxy_query_duration_milliseconds_bucket{query_type="UNKNOWN",outcome="bad_request",le="1.0"} 3
wdqs_proxy_query_duration_milliseconds_bucket{query_type="UNKNOWN",outcome="bad_request",le="+Inf"} 3
wdqs_proxy_query_duration_milliseconds_count{query_type="UNKNOWN",outcome="bad_request"} 3
wdqs_proxy_query_duration_milliseconds_sum{query_type="UNKNOWN",outcome="bad_request"} 2.0
# HELP wdqs_proxy_backend_duration_milliseconds SPARQL backend call latency
# TYPE wdqs_proxy_backend_duration_milliseconds histogram
wdqs_proxy_backend_duration_milliseconds_bucket{le="1.0"} 0
wdqs_proxy_backend_duration_milliseconds_bucket{le="5.0"} 2
wdqs_proxy_backend_duration_milliseconds_bucket{le="10.0"} 15
wdqs_proxy_backend_duration_milliseconds_bucket{le="50.0"} 98
wdqs_proxy_backend_duration_milliseconds_bucket{le="100.0"} 187
wdqs_proxy_backend_duration_milliseconds_bucket{le="500.0"} 239
wdqs_proxy_backend_duration_milliseconds_bucket{le="1000.0"} 244
wdqs_proxy_backend_duration_milliseconds_bucket{le="+Inf"} 244
wdqs_proxy_backend_duration_milliseconds_count 244
wdqs_proxy_backend_duration_milliseconds_sum 16901.0
# HELP wdqs_proxy_rewrite_errors_total SPARQL query rewrite errors
# TYPE wdqs_proxy_rewrite_errors_total counter
wdqs_proxy_rewrite_errors_total{error_type="parse_error"} 8.0
wdqs_proxy_rewrite_errors_total{error_type="federation_denied"} 3.0
wdqs_proxy_rewrite_errors_total{error_type="rewriter_error"} 1.0Comment Actions
Bucketing the latency in the application seems like the wrong place. Just report the actual query time and let the analytics tools handle the rest.
Comment Actions
gmodena updated https://gitlab.wikimedia.org/repos/wikidata-platform/wdqs/wdqs-proxy/-/merge_requests/31
Draft: service: add prometheus metrics reporting
Comment Actions
If we were to register the latency of each request as a metric, even at low QPS, the cardinality would explode due to high-cardinality labels and per-request values.
In Prometheus, the recommended approach is to expose a histogram and compute percentiles (p50/p95/p99) at query time (Grafana, alertmanager).
I experimented with a few approaches in the linked MR and settled on emitting Prometheus histograms, which should keep volume and cardinality reasonable (and bounded). We should still validate overall metrcis cardinality and volume with observability SRE to ensure it scales as expected (and we don't hammer their platform), but hopefully this is a start.
For analytics use cases, we will also log raw latency to the data lake for offline analysis and flexible re-bucketing.
Comment Actions
gmodena updated https://gitlab.wikimedia.org/repos/wikidata-platform/wdqs/wdqs-proxy/-/merge_requests/32
metrics: add prometheus dependencies and MetricsConfig
Comment Actions
Had a chat with @BTullis and we migth be able to customize bucket sizes for wdqs-proxy PODs in envoy. I'd rather do that than have ad-hoc (duplicated) metrics. I'll clean up the MR and drop bespoke latency reporting.
Comment Actions
gmodena merged https://gitlab.wikimedia.org/repos/wikidata-platform/wdqs/wdqs-proxy/-/merge_requests/32
metrics: add prometheus dependencies and MetricsConfig
Comment Actions
gmodena merged https://gitlab.wikimedia.org/repos/wikidata-platform/wdqs/wdqs-proxy/-/merge_requests/31
service: add custom metric reporting
Comment Actions
Relying on Envoy seems to be the right approach to avoid metric duplication. I'm moving the task to our radar. Feel free to reach out if you need anything else.