Page MenuHomePhabricator
Create Task
Maniphest T227668

Per-backend ATS Prometheus metrics
Closed, ResolvedPublic
Actions

Description

While investigating T184942: Deprecate python varnish cachestats we've ran into the fact that maps runs on cache upload, and the ATS migration for upload has been completed, and we don't have per-backend metrics (latency, status code, etc) from ATS. Ideally we have at least the same metrics we're collecting from varnishlog + mtail available from ATS, although I don't know the specifics of what's possible.

In terms of dashboards, we're looking at replacing varnish_backend_requests and varnish_backend_timing in dashboards and possibly alerts. The latter is a subset of the former, so we should be able to rewrite _timing in terms of _requests.

varnish_backend_requests

Matched db/api-frontend-summary (API frontend summary)
Matched db/maps-performances-filippo-t184942 (Maps performances Filippo T184942)
Matched db/wikidata-query-service-frontend (Wikidata Query Service Frontend)

varnish_backend_timing

Matched db/apache-backend-timing (Apache Backend-Timing)

Details

SubjectRepoBranchLines +/-
prometheus: add trafficserver_backend_requests_seconds_count rulesoperations/puppetproduction+5 -0
prometheus: rename trafficserver metricsoperations/puppetproduction+15 -15
prometheus: fetch ATS origin server metricsoperations/puppetproduction+14 -6
ATS: add atsbackend.mtailoperations/puppetproduction+70 -0
ATS: add support for atsmtail systemd servicesoperations/puppetproduction+30 -0
ATS: pass -socket and -regexp to fifo-log-taileroperations/puppetproduction+4 -1
0.3: implement fifo-log-tailer in gooperations/software/fifo-log-demuxmaster+48 -9
ATS: log origin server hostname and Backend-Timingoperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects

Event Timeline

fgiunchedi created this task.Jul 10 2019, 1:56 PM
Restricted Application added a project: SRE. · View Herald TranscriptJul 10 2019, 1:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• ema triaged this task as Medium priority.Jul 10 2019, 2:05 PM
• ema moved this task from Backlog to Caching on the Traffic board.
CDanis subscribed.Jul 10 2019, 2:07 PM
fgiunchedi mentioned this in T184942: Deprecate python varnish cachestats.Jul 11 2019, 9:45 AM
fgiunchedi added a project: User-fgiunchedi.Jul 12 2019, 8:50 AM
gerritbot added a comment.Jul 15 2019, 10:38 AM

Change 523130 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: log origin server hostname and Backend-Timing

https://gerrit.wikimedia.org/r/523130

gerritbot added a project: Patch-For-Review.Jul 15 2019, 10:38 AM
gerritbot added a comment.Jul 15 2019, 1:46 PM

Change 523168 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: add atsbackend.mtail

https://gerrit.wikimedia.org/r/523168

gerritbot added a comment.Jul 15 2019, 3:58 PM

Change 523130 merged by Ema:
[operations/puppet@production] ATS: log origin server hostname and Backend-Timing

https://gerrit.wikimedia.org/r/523130

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Jul 16 2019, 10:30 AM
gerritbot added a comment.Jul 16 2019, 1:02 PM

Change 523705 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: add support for atsmtail systemd services

https://gerrit.wikimedia.org/r/523705

gerritbot added a comment.Jul 16 2019, 4:39 PM

Change 523768 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/fifo-log-demux@master] 0.3: implement fifo-log-tailer in go

https://gerrit.wikimedia.org/r/523768

gerritbot added a comment.Jul 17 2019, 9:02 AM

Change 523768 merged by Ema:
[operations/software/fifo-log-demux@master] 0.3: implement fifo-log-tailer in go

https://gerrit.wikimedia.org/r/523768

Stashbot added a comment.Jul 17 2019, 9:07 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-17T09:07:44Z] <ema> upload fifo-log-demux 0.3 to stretch-wikimedia T227668

gerritbot added a comment.Jul 17 2019, 9:13 AM

Change 523881 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: pass -socket and -regexp to fifo-log-tailer

https://gerrit.wikimedia.org/r/523881

gerritbot added a comment.Jul 17 2019, 9:17 AM

Change 523881 merged by Ema:
[operations/puppet@production] ATS: pass -socket and -regexp to fifo-log-tailer

https://gerrit.wikimedia.org/r/523881

Stashbot added a comment.Jul 17 2019, 9:21 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-17T09:21:43Z] <ema> cp-ats: upgrade fifo-log-demux to 0.3 T227668

gerritbot added a comment.Jul 17 2019, 10:19 AM

Change 523705 merged by Ema:
[operations/puppet@production] ATS: add support for atsmtail systemd services

https://gerrit.wikimedia.org/r/523705

gerritbot added a comment.Jul 17 2019, 10:19 AM

Change 523168 merged by Ema:
[operations/puppet@production] ATS: add atsbackend.mtail

https://gerrit.wikimedia.org/r/523168

gerritbot added a comment.Jul 17 2019, 10:43 AM

Change 523898 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] prometheus: fetch ATS origin server metrics

https://gerrit.wikimedia.org/r/523898

fgiunchedi updated the task description. (Show Details)Jul 17 2019, 10:47 AM
gerritbot added a comment.Jul 17 2019, 12:35 PM

Change 523898 merged by Ema:
[operations/puppet@production] prometheus: fetch ATS origin server metrics

https://gerrit.wikimedia.org/r/523898

Stashbot added a comment.Jul 17 2019, 1:06 PM

Mentioned in SAL (#wikimedia-operations) [2019-07-17T13:06:47Z] <ema> prometheus servers: remove varnish-upload_$dc_backend.yaml, replaced by ATS equivalent T227668

Maintenance_bot removed a project: Patch-For-Review.Jul 17 2019, 1:11 PM
fgiunchedi mentioned this in T227432: Replace Varnish backends with ATS on cache text nodes.Jul 18 2019, 11:04 AM
gerritbot added a comment.Jul 23 2019, 1:05 PM

Change 525081 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] prometheus: add ats_backend_requests_seconds_count rules

https://gerrit.wikimedia.org/r/525081

gerritbot added a project: Patch-For-Review.Jul 23 2019, 1:05 PM
gerritbot added a comment.Jul 23 2019, 1:19 PM

Change 525085 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] prometheus: rename trafficserver metrics

https://gerrit.wikimedia.org/r/525085

gerritbot added a comment.Jul 23 2019, 2:33 PM

Change 525085 merged by Ema:
[operations/puppet@production] prometheus: rename trafficserver metrics

https://gerrit.wikimedia.org/r/525085

gerritbot added a comment.Jul 23 2019, 2:36 PM

Change 525081 merged by Ema:
[operations/puppet@production] prometheus: add trafficserver_backend_requests_seconds_count rules

https://gerrit.wikimedia.org/r/525081

Maintenance_bot removed a project: Patch-For-Review.Jul 23 2019, 3:10 PM
fgiunchedi added a comment.Aug 27 2019, 1:17 PM

Per-backend metrics are in place now via mtail, specifically:

  • request count: by backend, method, and status
  • total time spent took by requests: by backend, method, status
  • request latency buckets: by backend and method

Notably the buckets/histograms do not break down by status for cardinality reasons, although by sum()ing the _count and _sum metrics across status codes then the histogram_quantile functions can be used as expected

e.g.

cp1075:~$ curl -s localhost:3904/metrics | grep -v ^# | sort | grep -i appservers-rw
trafficserver_backend_requests_seconds_bucket{backend="appservers-rw.discovery.wmnet",le="+Inf",method="GET",prog="atsbackend.mtail"} 431060
trafficserver_backend_requests_seconds_bucket{backend="appservers-rw.discovery.wmnet",le="0.01",method="GET",prog="atsbackend.mtail"} 4931
trafficserver_backend_requests_seconds_bucket{backend="appservers-rw.discovery.wmnet",le="0.05",method="GET",prog="atsbackend.mtail"} 52582
trafficserver_backend_requests_seconds_bucket{backend="appservers-rw.discovery.wmnet",le="0.1",method="GET",prog="atsbackend.mtail"} 110478
trafficserver_backend_requests_seconds_bucket{backend="appservers-rw.discovery.wmnet",le="0.5",method="GET",prog="atsbackend.mtail"} 390573
trafficserver_backend_requests_seconds_bucket{backend="appservers-rw.discovery.wmnet",le="1.0",method="GET",prog="atsbackend.mtail"} 419147
trafficserver_backend_requests_seconds_bucket{backend="appservers-rw.discovery.wmnet",le="5.0",method="GET",prog="atsbackend.mtail"} 430660
trafficserver_backend_requests_seconds_count{backend="appservers-rw.discovery.wmnet",method="GET",status="200",prog="atsbackend.mtail"} 371045
trafficserver_backend_requests_seconds_count{backend="appservers-rw.discovery.wmnet",method="GET",status="204",prog="atsbackend.mtail"} 17315
trafficserver_backend_requests_seconds_count{backend="appservers-rw.discovery.wmnet",method="GET",status="301",prog="atsbackend.mtail"} 14349
trafficserver_backend_requests_seconds_count{backend="appservers-rw.discovery.wmnet",method="GET",status="302",prog="atsbackend.mtail"} 11916
trafficserver_backend_requests_seconds_count{backend="appservers-rw.discovery.wmnet",method="GET",status="303",prog="atsbackend.mtail"} 438
trafficserver_backend_requests_seconds_count{backend="appservers-rw.discovery.wmnet",method="GET",status="304",prog="atsbackend.mtail"} 4641
trafficserver_backend_requests_seconds_count{backend="appservers-rw.discovery.wmnet",method="GET",status="400",prog="atsbackend.mtail"} 353
trafficserver_backend_requests_seconds_count{backend="appservers-rw.discovery.wmnet",method="GET",status="403",prog="atsbackend.mtail"} 8
trafficserver_backend_requests_seconds_count{backend="appservers-rw.discovery.wmnet",method="GET",status="404",prog="atsbackend.mtail"} 10995
trafficserver_backend_requests_seconds_sum{backend="appservers-rw.discovery.wmnet",method="GET",status="200",prog="atsbackend.mtail"} 97109.5199999952
trafficserver_backend_requests_seconds_sum{backend="appservers-rw.discovery.wmnet",method="GET",status="204",prog="atsbackend.mtail"} 1215.767000000002
trafficserver_backend_requests_seconds_sum{backend="appservers-rw.discovery.wmnet",method="GET",status="301",prog="atsbackend.mtail"} 699.352999999978
trafficserver_backend_requests_seconds_sum{backend="appservers-rw.discovery.wmnet",method="GET",status="302",prog="atsbackend.mtail"} 1348.6029999999773
trafficserver_backend_requests_seconds_sum{backend="appservers-rw.discovery.wmnet",method="GET",status="303",prog="atsbackend.mtail"} 1.6779999999999828
trafficserver_backend_requests_seconds_sum{backend="appservers-rw.discovery.wmnet",method="GET",status="304",prog="atsbackend.mtail"} 255.75299999999888
trafficserver_backend_requests_seconds_sum{backend="appservers-rw.discovery.wmnet",method="GET",status="400",prog="atsbackend.mtail"} 34.54900000000002
trafficserver_backend_requests_seconds_sum{backend="appservers-rw.discovery.wmnet",method="GET",status="403",prog="atsbackend.mtail"} 0.424
trafficserver_backend_requests_seconds_sum{backend="appservers-rw.discovery.wmnet",method="GET",status="404",prog="atsbackend.mtail"} 2385.3479999999954
fgiunchedi moved this task from Doing to Backlog on the User-fgiunchedi board.Oct 18 2019, 1:40 PM
fgiunchedi moved this task from Backlog to Radar on the User-fgiunchedi board.Nov 13 2019, 11:23 AM
• ema closed this task as Resolved.Mar 16 2020, 3:56 PM
• ema claimed this task.
• ema subscribed.

Metrics added a while ago, closing!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits