Page MenuHomePhabricator
Create Task
Maniphest T399613

Finalize Charts SLO
Closed, ResolvedPublic
Actions

Description

https://wikitech.wikimedia.org/wiki/SLO/Charts is almost finalized, but not quite. We need to look at the data that we've gathered so far to get an idea for what target values would be realistic, then commit to those.

  • Decide on a maximum acceptable latency for the chart-renderer service. This will be the cutoff value in the combined latency-availability SLI for the service.
  • Decide on a target percentage for the latency-availability SLI for the service (what percentage of requests should complete faster than the cutoff value)
  • Decide on a target percentage for the availability SLI for client-side rendering
  • Add a graph for the combined latency-availability SLI to the Grafana dashboard
  • Update https://wikitech.wikimedia.org/wiki/SLO/Charts with these values and mark it as complete
  • Define the SLOs in Pyrra

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
pyrra: add the Charts SLOsoperations/puppetproduction+57 -0
Customize query in gerrit

Related Objects

Event Timeline

Catrope created this task.Jul 15 2025, 5:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 15 2025, 5:14 PM
egardner subscribed.Jul 15 2025, 5:15 PM
egardner added a comment.Jul 15 2025, 5:34 PM

Should this be a Reader Growth Team task instead?

ovasileva edited projects, added Reader Growth Team; removed Reader Experience Team.Jul 24 2025, 11:37 AM
elukey subscribed.Jul 28 2025, 10:42 AM

@Catrope Hi! Any update on the timeline for this task? :)

elukey added a comment.Aug 1 2025, 3:07 PM

@egardner Hi! I see that Roan is on holidays, do you have any update on this? Thanks in advance :)

MBinder_WMF added a subscriber: NBaca-WMF.Aug 1 2025, 5:20 PM
MBinder_WMF subscribed.
ovasileva triaged this task as Medium priority.Aug 4 2025, 4:15 PM
Catrope updated the task description. (Show Details)Aug 13 2025, 5:36 AM
egardner moved this task from Incoming/Inbox to Needs Refinement on the Reader Growth Team board.Aug 14 2025, 4:48 PM
elukey added a comment.Sep 2 2025, 2:19 PM

Getting back to this, next step is adding the Pyrra config. We are experimenting with a new revision-based config, so I may wait a couple of days more to avoid the transition of Charts to it :)

To recap, these are the SLIs:

chart-renderer service: Combined latency-availability

The percentage of all requests that complete within 200 milliseconds and receive a non-error response (HTTP 200).

In the dashboard the relevant graph seems to be this one, but I am not sure where the metric comes from (namely, where it is emitted from). The chart-renderer service is deployed in k8s and it is behind the Istio Gateway, we usually use these metrics for Pyrra. What do you think?

Chart client side rendering: Availability

The percentage of client-side rendering attempts that successfully display a chart

In this case the metrics to use seem to be mediawiki_Chart_render_failure_total and mediawiki_Chart_render_end_total, am I correct?

Catrope added a subscriber: CDanis.Sep 2 2025, 9:43 PM
In T399613#11139447, @elukey wrote:

chart-renderer service: Combined latency-availability

The percentage of all requests that complete within 200 milliseconds and receive a non-error response (HTTP 200).

In the dashboard the relevant graph seems to be this one, but I am not sure where the metric comes from (namely, where it is emitted from). The chart-renderer service is deployed in k8s and it is behind the Istio Gateway, we usually use these metrics for Pyrra. What do you think?

I have no idea where either of those metrics come from or how they are different. I think @CDanis might be the one who set up the dashboard originally so they might know. My guess is that the Istio metrics you linked to are probably fine.

Chart client side rendering: Availability

The percentage of client-side rendering attempts that successfully display a chart

In this case the metrics to use seem to be mediawiki_Chart_render_failure_total and mediawiki_Chart_render_end_total, am I correct?

Yes, that is correct. Specifically, render_failure_total is the number of failed renders, and render_end_total is the number of successful renders, ao the success percentage is render_end_total / (render_end_total + render_failure_total). That formula is used in the Grafana chart too (except that it graphs the failure percentage instead).

Jdlrobson-WMF closed this task as Resolved.Sep 10 2025, 4:53 PM
Jdlrobson-WMF claimed this task.
Jdlrobson-WMF mentioned this in T404222: [Spike] Create task with follow up for charts metrics to finalize SLO.
Jdlrobson-WMF subscribed.

We'll work out remaining work here using the spike in T404222: [Spike] Create task with follow up for charts metrics to finalize SLO

Catrope reopened this task as Open.Sep 10 2025, 8:08 PM
Catrope reassigned this task from Jdlrobson-WMF to elukey.
Catrope moved this task from Needs Refinement to Radar on the Reader Growth Team board.

The only remaining work here is for @elukey to configure the SLO in Pyrra, so reopening this task and assigning to him.

Catrope added a project: SRE-SLO.Sep 10 2025, 8:40 PM
elukey added a comment.Sep 16 2025, 3:26 PM

Thanks a lot for the patience folks, we have stopped onboarding new SLOs in Pyrra temporarily while we figure out T403729. We are comparing the results with another tool in T404171, so we are trying to avoid onboarding something on Pyrra to then do it again on the new tool. I hope that this will be solved soon!

CDanis added a comment.Sep 16 2025, 3:36 PM

Luca, do you want an early test subject for the Sloth trial?

elukey added a comment.Sep 16 2025, 3:50 PM
In T399613#11185664, @CDanis wrote:

Luca, do you want an early test subject for the Sloth trial?

Definitely, the first use case will be Citoid so we can make a comparison with Pyrra, but Charts could definitely be the second one!

gerritbot added a comment.Sep 23 2025, 10:23 AM

Change #1190620 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] pyrra: add the Charts SLOs

https://gerrit.wikimedia.org/r/1190620

gerritbot added a project: Patch-For-Review.Sep 23 2025, 10:23 AM
gerritbot added a comment.Sep 24 2025, 7:17 AM

Change #1190620 merged by Elukey:

[operations/puppet@production] pyrra: add the Charts SLOs

https://gerrit.wikimedia.org/r/1190620

Maintenance_bot removed a project: Patch-For-Review.Sep 24 2025, 7:32 AM
elukey added a comment.Sep 24 2025, 8:51 AM

Hi folks! I created two dashboards for charts:

Rolling windows: https://slo.wikimedia.org/?search=charts
Calendar windows:

More info about those in:
https://wikitech.wikimedia.org/wiki/SLO/Template_instructions/Dashboards_and_alerts
https://wikitech.wikimedia.org/wiki/SLO/Template_instructions/Dashboards_and_alerts#How_to_read_a_Pyrra_dashboard

Even if Pyrra shows a different behavior than expected in some places, we decided to keep going with it for a little while more to gather some feedback :)

In my opinion when the dashboards are settled (namely some data is visualized etc..) we can probably declare the SLOs approved!

CDanis closed this task as Resolved.Oct 3 2025, 4:50 PM

Dashboards look good to me, let's call this finalized!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits