Page MenuHomePhabricator
Create Task
Maniphest T190992

prometheus: slow dashboards due to suboptimal query_range performance
Closed, ResolvedPublic
Actions

Description

We have ported several dashboards from graphite to prometheus, including Aggregate Client Status Code. While the graphite version loads quickly and feels snappy, the prometheus one is quite slow.

While inspecting the situation using chrome developer tools, I've noticed that certain query_range requests to the prometheus API have a TTFB between 7 and 11 seconds.

See for example this query.

My initial thought was to write some aggregation rules to speed things up, but the dashboard is already using job_method_status:varnish_requests:rate5m.

Related Objects
Search...

Event Timeline

• ema created this task.Mar 29 2018, 8:23 AM
Restricted Application added a project: SRE. · View Herald TranscriptMar 29 2018, 8:23 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• ema triaged this task as Medium priority.Mar 29 2018, 8:23 AM
• ema moved this task from Backlog to Radar/Not for service by Traffic on the Traffic board.Mar 29 2018, 8:25 AM
CDanis mentioned this in T212312: prometheus-based graph significantly slower than statsd equivalent.Jan 9 2019, 3:26 PM
CDanis added a parent task: T187987: 100% of Prometheus traffic served by Prometheus v2.
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:53 PM
Volans subscribed.Apr 1 2019, 3:20 PM

@ema given the speedup due to prometheus 2 do you think this still needs to be worked on or could be resolved?

• ema closed this task as Resolved.Apr 1 2019, 4:00 PM
• ema claimed this task.
In T190992#5074344, @Volans wrote:

@ema given the speedup due to prometheus 2 do you think this still needs to be worked on or could be resolved?

Ah yes, thanks for the ping. Closing.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL