| bking | |
| Feb 23 2026, 8:52 PM |
| F75846572: image.png | |
| Apr 13 2026, 1:47 PM |
Per parent ticket, we are moving our first tenant workload for the OpenSearch on K8s platform into production.
We should create an SLO for this cluster.
Creating this ticket to:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| ipod: fix opensearch availability and latency SLI definitions | repos/sre/slothslos!28 | elukey | ipoid_sli_fix | main | |
| Add SLOs for opensearch-ipoid availability and latency | repos/sre/slothslos!18 | btullis | ipoid | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T362105 EPIC: OpenSearch on K8s (formerly Mutualized opensearch cluster) - FY25/26 WE4.2.6 | |||
| Resolved | bking | T408586 ☂️ OpenSearch on K8s: Ensure that our first tenant workload is ready for production ☂️ | |||
| Resolved | kostajh | T325147 New Service Request 'iPoid' | |||
| Resolved | kostajh | T348935 IPoid: Define service level indicators and service level objectives | |||
| Open | Gehel | T418175 Create SLO for the opensearch-ipoid cluster that runs on our OpenSearch on K8s platform | |||
| Resolved | bking | T419289 Ensure OpenSearch on k8s clusters can safely use envoy TLS termination | |||
| In Progress | bking | T422378 Move OpenSearch-iPoid behind TLS/Update to OpenSearch 3.x |
I am struggling a little with the concept of what we are trying to measure here.
We already have a draft SLO in the pipeline for OpenSearch IPoid, which is our first production tenant service running under Kubernetes.
There are some SLIs already defined for that service, and we can see that all of the required metrics are already visible on the OpenSearch on K8S dashboard, when selecting this tenant cluster.
I think that the work for this is already being handled under: T348935: IPoid: Define service level indicators and service level objectives although I may be wrong.
However, I'm a bit vague on what this ticket requires in terms of an SLO for the OpenSearch on K8S platform, as distinct from any of the opensearch clusters that it manages.
I have noticed that we are currently not scraping any metrics from the opensearch-operator, so we should collect those, but I'm not immediately sure how we would buil;d a useful SLO from them.
Change #1248865 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/deployment-charts@master] opensearch-cluster: Terminate TLS on the ingress gateway
After a discussion with @Gehel I now have much greater clarity on what is required.
It is an SLO for the tenant workload that we currently, but it should be a sort of template for an SLO that could be applied with minimal modification to any opensearch on k8s workload.
As far as possible, it should be relatively self-contained, such that the SLIs are built on metrics that are created within the opensearch on k8s platform, as opposed to instrumentation of the client application.
This will help in making sure that the SLO template can easily be applied to a different cluster.
Availability metrics should be relatively uniform across all opensearech clusters, but latency metrics might differ markedly, based on the workload of the cluster.
I have created: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1248865 which I believe will enable the ingressgateway to generate metrics about HTTP responses and latencies that we can use.
Change #1248865 merged by Bking:
[operations/deployment-charts@master] opensearch-cluster: Terminate TLS on the ingress gateway
Change #1266935 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] deployment_server: monitor the expirty of the internal opensearch TLS certs
Change #1266935 merged by Brouberol:
[operations/puppet@production] deployment_server: monitor the expiry of the internal opensearch TLS certs
Change #1266999 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] deployment_server: tweak the labels on opensearch_k8s_master_cert_expiry_seconds
Change #1266999 merged by Brouberol:
[operations/puppet@production] deployment_server: tweak the labels on opensearch_k8s_master_cert_expiry_seconds
Change #1267100 had a related patch set uploaded (by Bking; author: Bking):
[operations/alerts@master] data-platform: Add alerts for opensearch on k8s certificate expiry
I'm reassigning this to you for now @bking - as I'll be out next week and you've been making solid progress on the telemetry work. Hope that's OK.
Change #1267100 merged by Bking:
[operations/alerts@master] data-platform: Add alerts for opensearch on k8s certificate expiry
No problem! I'm blocking this ticket on T422378 since we have determined we need the Envoy telemetry metrics to properly define the SLO.
We've changed opensearch-ipoid to terminate at the ingress gateway instead of the pod level. Unfortunately, this change is not enough on its own to get the metrics we crave.
This comment by @JMeybohm describes the situation we are in:
If you want mesh support for this (client -> envoy -> envoy -> opensearch) you will have to add the tls terminating sidecar container (like we do with all k8s deployments) as well as a listener definition from Ia48de6243cbea3bf5296bc06c0119f4bb64073c1.
If you're terminating TLS differently, you might get away with just the latter. But it will make this implementation a snowflake and hard to debug for outsiders. Or better put: It will probably send people down some rabbit hole since they will expect the service mesh envoy (it's metrics and logs) on both ends of the communication.
At this point, the bare minimum we need to do is to finish the listener definition change above. That should give us metrics from at least one side of the connection, and since iPoid only has one client (mediawiki) they should be sufficiently representative.
Adding the TLS terminator as suggested requires us to further customize the current chart (which we're trying to get off of anyway, ref T417187). I prefer to wait until we are on the newer chart before tackling that problem.
Reassigning this back to myself.
There was a little confusion, but I'm confident that we have the metrics we need to proceed.
The Istio dashboard shows that we can now get both the availability and latency metrics from the ingressgateway, now that we are terminating TLS using the SIMPLE method, rather than the previous PASSTHROUGH method.
https://grafana.wikimedia.org/goto/ffiyy5jaatkowf?orgId=1
It wasn't until we had merged https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1248865 when we enabled this for all opensearch clusters.
So now we have:
client -> envoy (istio) -> opensearch
..and the metrics we need can be obtained from istio now, at the expense of possibly another network hop.
I'll start trying to update the SLI definition now and make progress on the SLO.
Change #1277486 had a related patch set uploaded (by Btullis; author: Btullis):
[operations/deployment-charts@master] opensearch-cluster: Istio configure bulk indexing service
Change #1277486 merged by jenkins-bot:
[operations/deployment-charts@master] opensearch-cluster: Istio configure bulk indexing service
The sloth definitions for the SLOs are here: https://gitlab.wikimedia.org/repos/sre/slothslos/-/merge_requests/18
I have requested a review of both the SLOs and the sloth configuration by the SRE-SLO working group.
In the meantime, I'll mark this ticket as in-review.
There is a conversation about this SLO in the #slos Slack channel.
The advice from @elukey is as follows:
The freshness SLI for Airflow is a bit tricky, since as you mentioned we don't have a clear time series for it (https://wikitech.wikimedia.org/wiki/SLO/MediaWiki_Content_History_Table comes to mind as similar).
I would personally leave it aside for the first step, and concentrate on the other two SLIs. We can revisit and come back later...
I'm happy enough to do this, in principle.
maybe we could add some more instrumentation to the Airflow job that would allow us to generate this proper time-series for a freshness SLI.
In the meantime, I will remove the freshness SLO from the sloth patch and seek another review.
elukey merged https://gitlab.wikimedia.org/repos/sre/slothslos/-/merge_requests/18
Add SLOs for opensearch-ipoid availability and latency
The patch to implement the SLOs has been merged, so we now have graphs.
https://grafana.wikimedia.org/goto/efmqukgcvfda8a?orgId=1
I'll check back over the next few days to ascertain whether everything is working correctly, or it needs further work.
Hi, can I confirm that https://wikitech.wikimedia.org/wiki/SLO/ipoid should now redirect to https://wikitech.wikimedia.org/wiki/SLO/OpenSearch_IPoid ?
I'll tentatively resolve this ticket, since we have the SLOs in place and the dashboards are set up here: https://grafana.wikimedia.org/goto/cfmuecngqz8jkb?orgId=1
One thing to note is that https://wikitech.wikimedia.org/wiki/SLO/OpenSearch_IPoid is still marked as draft.
I'm not sure who would like to update this status and when. Perhaps we need to wait for 30 days or a quarter to make sure that the SLO values are correct, or perhaps we can update it now.
Feel free to reassign and/or reopen the ticket if you feel that there is more work to be done.
Let's leave it open until we move it from "draft" to "approved." :)
You can find notes on how to do that at wt:SLO/Runbook#Publication but it's pretty straightforward. If both PSI and DP SRE -- that is, the owning teams listed on the SLO, who might need to do work to support it -- are ready to commit to it, then you can just sign it off on-wiki and close this. If you'd like to wait a few days and collect some data first, you absolutely can, but a quarter is almost certainly longer than you need.
Even after exiting draft status, you can still make changes! It just means, if it breaks, you officially promise to fix it.
The dashboard has some problems, I see one of the rolling window displaying an error budget of -1900% and the other one already below 0%. There may be some issues in the SLI metrics, I'll check on Monday :)
elukey merged https://gitlab.wikimedia.org/repos/sre/slothslos/-/merge_requests/28
ipod: fix opensearch availability and latency SLI definitions
Per my last comment I'd still like to see the SLO moved out of draft status before we resolve the ticket, just so we don't forget to do it.
I've edited the page to reflect the latest changes to the slothslos configuration for opensearch-ipoid and to reflect the change to the request latency to be 500ms.
Reassigning to @Gehel for final review and publication, as long as no further changes are required at this stage.
@Rsilvola / @OKryva-WMF : could you sign off on the SLO by adding a comment on this task?