Page MenuHomePhabricator
Create Task
Maniphest T209108

Monitoring and data collection for session storage service
Closed, ResolvedPublic
Actions

Description

The session storage service will need to export a sensible set of metrics. Grafana dashboards also need to be created that visualize data from these metrics and provide operational insight.

Additionally (as is convention for services hosted at the WMF), Icinga checks that perform a synthetic transaction according to a (discovered) specification should be possible.

See also: operations/software/service-checker

Details

SubjectRepoBranchLines +/-
prometheus: collect session storage Cassandra metricsoperations/puppetproduction+6 -0
Implement a `/healthz` endpoint for k8s readiness probemediawiki/services/kaskmaster+16 -0
HTTP Prometheus metricsmediawiki/services/kaskmaster+29 -2
Basic prometheus metrics supportmediawiki/services/kaskmaster+4 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Eevans triaged this task as Medium priority.Nov 8 2018, 8:45 PM
Eevans created this task.
gerritbot subscribed.Jan 14 2019, 11:05 PM

Change 484315 had a related patch set uploaded (by Eevans; owner: Eevans):
[mediawiki/services/kask@master] Basic prometheus metrics support

https://gerrit.wikimedia.org/r/484315

gerritbot added a project: Patch-For-Review.Jan 14 2019, 11:05 PM
Eevans mentioned this in rMSKSb2a6db117bad: Basic prometheus metrics support.Jan 14 2019, 11:07 PM
Eevans added a comment.Jan 15 2019, 1:45 PM

For posterity sake, here are the Prometheus metrics provided by default:

P7990 Masterwork From Distant Lands
1HTTP/1.1 200 OK
2Content-Length: 5007
3Content-Type: text/plain; version=0.0.4
4Date: Tue, 15 Jan 2019 13:44:03 GMT
5
6# HELP go_gc_duration_seconds A summary of the GC invocation durations.
7# TYPE go_gc_duration_seconds summary
8go_gc_duration_seconds{quantile="0"} 0
9go_gc_duration_seconds{quantile="0.25"} 0
10go_gc_duration_seconds{quantile="0.5"} 0
11go_gc_duration_seconds{quantile="0.75"} 0
12go_gc_duration_seconds{quantile="1"} 0
13go_gc_duration_seconds_sum 0
14go_gc_duration_seconds_count 0
15# HELP go_goroutines Number of goroutines that currently exist.
16# TYPE go_goroutines gauge
17go_goroutines 13
18# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
19# TYPE go_memstats_alloc_bytes gauge
20go_memstats_alloc_bytes 900928
21# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
22# TYPE go_memstats_alloc_bytes_total counter
23go_memstats_alloc_bytes_total 900928
24# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
25# TYPE go_memstats_buck_hash_sys_bytes gauge
26go_memstats_buck_hash_sys_bytes 2473
27# HELP go_memstats_frees_total Total number of frees.
28# TYPE go_memstats_frees_total counter
29go_memstats_frees_total 500
30# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
31# TYPE go_memstats_gc_sys_bytes gauge
32go_memstats_gc_sys_bytes 131072
33# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
34# TYPE go_memstats_heap_alloc_bytes gauge
35go_memstats_heap_alloc_bytes 900928
36# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
37# TYPE go_memstats_heap_idle_bytes gauge
38go_memstats_heap_idle_bytes 57344
39# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
40# TYPE go_memstats_heap_inuse_bytes gauge
41go_memstats_heap_inuse_bytes 1.581056e+06
42# HELP go_memstats_heap_objects Number of allocated objects.
43# TYPE go_memstats_heap_objects gauge
44go_memstats_heap_objects 8709
45# HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.
46# TYPE go_memstats_heap_released_bytes_total counter
47go_memstats_heap_released_bytes_total 0
48# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
49# TYPE go_memstats_heap_sys_bytes gauge
50go_memstats_heap_sys_bytes 1.6384e+06
51# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
52# TYPE go_memstats_last_gc_time_seconds gauge
53go_memstats_last_gc_time_seconds 0
54# HELP go_memstats_lookups_total Total number of pointer lookups.
55# TYPE go_memstats_lookups_total counter
56go_memstats_lookups_total 39
57# HELP go_memstats_mallocs_total Total number of mallocs.
58# TYPE go_memstats_mallocs_total counter
59go_memstats_mallocs_total 9209
60# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
61# TYPE go_memstats_mcache_inuse_bytes gauge
62go_memstats_mcache_inuse_bytes 4800
63# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
64# TYPE go_memstats_mcache_sys_bytes gauge
65go_memstats_mcache_sys_bytes 16384
66# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
67# TYPE go_memstats_mspan_inuse_bytes gauge
68go_memstats_mspan_inuse_bytes 27360
69# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
70# TYPE go_memstats_mspan_sys_bytes gauge
71go_memstats_mspan_sys_bytes 32768
72# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
73# TYPE go_memstats_next_gc_bytes gauge
74go_memstats_next_gc_bytes 4.194304e+06
75# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
76# TYPE go_memstats_other_sys_bytes gauge
77go_memstats_other_sys_bytes 1.066583e+06
78# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
79# TYPE go_memstats_stack_inuse_bytes gauge
80go_memstats_stack_inuse_bytes 458752
81# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
82# TYPE go_memstats_stack_sys_bytes gauge
83go_memstats_stack_sys_bytes 458752
84# HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.
85# TYPE go_memstats_sys_bytes gauge
86go_memstats_sys_bytes 3.346432e+06
87# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
88# TYPE process_cpu_seconds_total counter
89process_cpu_seconds_total 0.02
90# HELP process_max_fds Maximum number of open file descriptors.
91# TYPE process_max_fds gauge
92process_max_fds 1024
93# HELP process_open_fds Number of open file descriptors.
94# TYPE process_open_fds gauge
95process_open_fds 11
96# HELP process_resident_memory_bytes Resident memory size in bytes.
97# TYPE process_resident_memory_bytes gauge
98process_resident_memory_bytes 8.876032e+06
99# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
100# TYPE process_start_time_seconds gauge
101process_start_time_seconds 1.54755983088e+09
102# HELP process_virtual_memory_bytes Virtual memory size in bytes.
103# TYPE process_virtual_memory_bytes gauge
104process_virtual_memory_bytes 4.645888e+08

fgiunchedi subscribed.Jan 15 2019, 1:50 PM

Documentation pointers on Prometheus' data model, metric types, etc: https://prometheus.io/docs/concepts/data_model/ https://prometheus.io/docs/concepts/metric_types/ https://prometheus.io/docs/practices/naming/

Also wrt percentiles there's a change in how we should aggregate timings across service instances, compared to statsd, I've started documenting it here (in the context of k8s but applies generally) and feedback/improvements are very much welcome https://wikitech.wikimedia.org/wiki/Prometheus/statsd_k8s#Global_aggregation_considerations

Joe subscribed.Jan 15 2019, 1:54 PM
Eevans mentioned this in rMSKS443f8ceffbc9: Basic prometheus metrics support.Jan 15 2019, 5:53 PM
gerritbot added a comment.Jan 15 2019, 9:43 PM

Change 484315 merged by Eevans:
[mediawiki/services/kask@master] Basic prometheus metrics support

https://gerrit.wikimedia.org/r/484315

gerritbot added a comment.Jan 25 2019, 5:52 PM

Change 486502 had a related patch set uploaded (by Clarakosi; owner: Clarakosi):
[mediawiki/services/kask@master] HTTP Prometheus metrics

https://gerrit.wikimedia.org/r/486502

• Clarakosi mentioned this in rMSKS7b91adaaf6e8: HTTP Prometheus metrics.Jan 25 2019, 5:56 PM
• Clarakosi mentioned this in rMSKSaa9770a19393: HTTP Prometheus metrics.Feb 5 2019, 3:33 PM
• Clarakosi mentioned this in rMSKS98e5250da504: HTTP Prometheus metrics.Feb 5 2019, 4:52 PM
• Clarakosi mentioned this in rMSKS64cfdbe1a6c7: HTTP Prometheus metrics.Feb 5 2019, 5:05 PM
Eevans mentioned this in rMSKS7991ff90b83a: HTTP Prometheus metrics.Feb 5 2019, 8:10 PM
gerritbot added a comment.Feb 5 2019, 8:21 PM

Change 486502 merged by Eevans:
[mediawiki/services/kask@master] HTTP Prometheus metrics

https://gerrit.wikimedia.org/r/486502

• Clarakosi moved this task from Backlog to In-Progress on the User-Clarakosi board.Feb 8 2019, 5:43 PM
gerritbot added a comment.Feb 27 2019, 3:30 PM

Change 493249 had a related patch set uploaded (by Eevans; owner: Eevans):
[mediawiki/services/kask@master] Implement a /healthz endpoint for k8s readiness probe

https://gerrit.wikimedia.org/r/493249

Eevans mentioned this in rMSKS9fbf8f34eca5: Implement a `/healthz` endpoint for k8s readiness probe.Feb 27 2019, 3:32 PM
Eevans mentioned this in rMSKS6f6662e40332: Implement a `/healthz` endpoint for k8s readiness probe.Mar 5 2019, 7:32 PM
gerritbot added a comment.Mar 5 2019, 7:32 PM

Change 493249 merged by Clarakosi:
[mediawiki/services/kask@master] Implement a /healthz endpoint for k8s readiness probe

https://gerrit.wikimedia.org/r/493249

Eevans renamed this task from Metrics for session storage service to Monitoring and data collection for session storage service.Mar 7 2019, 5:10 PM
Eevans updated the task description. (Show Details)
Eevans added subscribers: jijiki, akosiaris.
jijiki mentioned this in rMSKSd0b02bcd3e8e: HTTP Prometheus metrics.Mar 18 2019, 4:35 PM
jijiki mentioned this in rMSKS103c39823326: Basic prometheus metrics support.
gerritbot added a comment.Mar 20 2019, 7:21 PM

Change 497848 had a related patch set uploaded (by Eevans; owner: Eevans):
[operations/puppet@production] prometheus: collect session storaage Cassandra metrics

https://gerrit.wikimedia.org/r/497848

gerritbot added a comment.Mar 27 2019, 2:29 PM

Change 497848 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: collect session storage Cassandra metrics

https://gerrit.wikimedia.org/r/497848

Eevans mentioned this in T206015: Plan/design a session storage service.Apr 1 2019, 2:23 PM
Eevans moved this task from Backlog to In-Progress on the User-Eevans board.Apr 25 2019, 8:17 PM
gerritbot added a comment.Apr 30 2019, 6:30 PM

Change 507397 had a related patch set uploaded (by Eevans; owner: Eevans):
[mediawiki/services/kask@master] [WIP] Serve OpenAPI specification

https://gerrit.wikimedia.org/r/507397

Eevans mentioned this in rMSKS62a132898024: [WIP] Serve OpenAPI specification.Apr 30 2019, 6:52 PM
Eevans closed subtask T222227: Kask support for operations/software/service-checker as Resolved.May 10 2019, 4:27 PM
Eevans added a comment.May 30 2019, 5:26 PM

https://grafana.wikimedia.org/d/000001590/sessionstore?refresh=1m&orgId=1 is now available courtesy of @akosiaris

Eevans closed this task as Resolved.Jun 11 2019, 6:19 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL