Page MenuHomePhabricator
Create Task
Maniphest T263536

Aggregated metrics for ats-tls <-> clients ttfb percentiles
Closed, ResolvedPublic
Actions

Description

We have metrics for ats-tls TTFB for individual instances, but not yet aggregated per cluster/site. The end goal is to expose such metrics in "frontend traffic"-like dashboards and thus observe aggregated TTFB at the edge.

Relevant metrics:

trafficserver_tls_client_ttfb_bucket
trafficserver_tls_client_ttfb_sum
trafficserver_tls_client_ttfb_count

Details

SubjectRepoBranchLines +/-
prometheus: add 50 percentile for ats-tls TTFBoperations/puppetproduction+2 -0
prometheus: aggregation rules for ats-tls client TTFBoperations/puppetproduction+14 -0
Customize query in gerrit

Related Objects

Event Timeline

fgiunchedi created this task.Sep 22 2020, 12:28 PM
Restricted Application added a project: SRE. · View Herald TranscriptSep 22 2020, 12:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
CDanis subscribed.Sep 22 2020, 12:58 PM
gerritbot added a comment.Sep 23 2020, 4:19 PM

Change 629430 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: aggregation rules for ats-tls client TTFB

https://gerrit.wikimedia.org/r/629430

gerritbot added a project: Patch-For-Review.Sep 23 2020, 4:19 PM
• crusnov assigned this task to fgiunchedi.Sep 23 2020, 7:01 PM
• crusnov triaged this task as Medium priority.
gerritbot added a comment.Sep 28 2020, 9:53 AM

Change 629430 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: aggregation rules for ats-tls client TTFB

https://gerrit.wikimedia.org/r/629430

fgiunchedi added a project: User-fgiunchedi.Sep 28 2020, 9:58 AM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.
Maintenance_bot removed a project: Patch-For-Review.Sep 28 2020, 10:11 AM
fgiunchedi moved this task from Inbox to In progress on the observability board.Sep 28 2020, 3:10 PM
fgiunchedi added a comment.Oct 1 2020, 11:02 AM

Added a panel to https://grafana.wikimedia.org/d/000000479/frontend-traffic to showcase the top p95 offenders:

2020-10-01-130213_1203x320_scrot.png (320×1 px, 87 KB)

I'm sure there is optimization/tweaking to do, let me know what you think!

gerritbot added a comment.Oct 5 2020, 8:19 AM

Change 632190 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: add 50 percentile for ats-tls TTFB

https://gerrit.wikimedia.org/r/632190

gerritbot added a project: Patch-For-Review.Oct 5 2020, 8:19 AM
gerritbot added a comment.Oct 6 2020, 11:05 AM

Change 632190 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: add 50 percentile for ats-tls TTFB

https://gerrit.wikimedia.org/r/632190

fgiunchedi added a comment.Oct 6 2020, 4:12 PM

With 50 percentile added I'm considering this closed!

As a demo/playground I've started https://grafana.wikimedia.org/d/fD5qC_KGz/filippo-ats-latency for anyone interested

fgiunchedi closed this task as Resolved.Oct 6 2020, 4:13 PM
fgiunchedi added a comment.Oct 13 2020, 10:40 AM
In T263536#6522137, @fgiunchedi wrote:

With 50 percentile added I'm considering this closed!

As a demo/playground I've started https://grafana.wikimedia.org/d/fD5qC_KGz/filippo-ats-latency for anyone interested

Moved to https://grafana.wikimedia.org/d/8T2XA-5Gz/frontend-ats-tls-ttfb-latency

fgiunchedi mentioned this in T253655: Document and/or improve navigation of the various HTTP frontend Grafana dashboards.Oct 13 2020, 10:40 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL