Per this Slack thread , the opensearch-ipoid API endpoint sometimes responds more slowly than expected.
Creating this ticket to:
A few notes on the investigation:
The blackbox probes (HTTP health checks initiated by prometheus) go to a different prometheus instance (ops, as opposed to dse-k8s) than the rest of the OpenSearch metrics.
Thus, to get a historical picture of latency from Prometheus, we need to look in the ops instance, using a PromQL query like probe_success{job="probes/service",module=~"http_opensearch-.*"}
I'm working on getting this into the OpenSearch on K8s dashboard .
To get an idea of latency in ms, I think we can use the query
avg_over_time(probe_duration_seconds{job="probes/service",module=~"http_opensearch-.*"}[2m])* 1000I've added blackbox probe latency panels to the OpenSearch on K8s dashboard.
From this dashboard we can see that the recorded latency is considerably higher in EQIAD than in CODFW. This is also reflected in the OpenSearch on iPoid dashboard , which sources its metrics from MediaWiki.
I'm not sure what to make of this yet, but I will keep investigating. Note that my very unscientific ping tests between DCs suggest about ~30ms latency, so I don't think it's a simple matter of metrics being scraped from the wrong DC.
Change #1236818 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] opensearch on k8s: enable blackbox probes for opensearch-test ns
Change #1236818 merged by Bking:
[operations/puppet@production] opensearch on k8s: enable blackbox probes for opensearch-test ns
Change #1237303 had a related patch set uploaded (by Bking; author: Bking):
[operations/deployment-charts@master] opensearch-ipoid: add cert SANs for non-discovery endpoints
Change #1237303 abandoned by Bking:
[operations/deployment-charts@master] DO NOT MERGE: opensearch-ipoid: add cert SANs for non-discovery endpoints
Reason:
no longer needed
I've created another dashboard, DSE K8s Blackbox probes , to aid the investigation.
Any dse-k8s-hosted service that is listed as "production" in the Puppet service definition file should be visible in the dashboard.
We do see some spikiness and increased latency on most services in EQIAD, which is a bit confusing, but for the most part it does not seem like it should be 240ms to go from MW to OpenSearch, as @kostajh mentioned in the Slack thread. We also don't know which DC or DCs these probes originate from, and we should probably understand that better before moving forward.
We've also gotten another ProbeSlow alert for opensearch-ipoid, which fired at at 17:39:39 and resolved exactly 5 minutes later. This doesn't seem to be reflected on the iPoid dashboard, but it uses bucketing so it must not have been disruptive enough to affect p99.
Also, it's a long shot, but I also noticed some chatter in #wikimedia-irc today about suboptimal routing for hCaptcha and I thought maybe the same thing could be happening here? Will ask on Monday. Update: This turned out to be irrelevant.
The new opensearch backend rolled out today from ~1145-1245 UTC. It had to be reverted due to timeouts.
I'm investigating now and will update the ticket when I have more info.
Change #1238409 had a related patch set uploaded (by Bking; author: Bking):
[operations/deployment-charts@master] opensearch-on-k8s: avoid k8s workers with 1GB NIC
Change #1238409 merged by jenkins-bot:
[operations/deployment-charts@master] opensearch-on-k8s: avoid k8s workers with 1GB NIC
Change #1238441 had a related patch set uploaded (by Bking; author: Bking):
[operations/dns@master] dse-k8s: Enable active/active for dse-k8s clusters
Mentioned in SAL (#wikimedia-operations) [2026-02-10T23:04:15Z] <ryankemper> [opensearch-ipoid] T416345 Restarted (~16 mins ago) opensearch-ipoid-masters-1 to force it to schedule to a k8s node with 10G networking, hopefully this helps w/ latency
Update: One of the OpenSearch pods has significantly higher search latency than the other 2. The slower pod is on 1Gbps network host.
We've previously seen increased pod latency due to 1Gbps connections in T415635, and there seems to be a strong correlation here as well. After redeploying opensearch-ipoid-masters-1 on a 10G host, latency seems to have dropped.
Note that a helmfile apply was not sufficient to apply the new anti-affinity settings; we also had to `kubectl edit opensearchclusters.
opensearch.opster.io` and add the affinity settings manually.
There is also a separate issue where opensearch-ipoid.discovery.wmnet seems to be resolved correctly (different IP depending on which DC I'm resolving it from), but the API call to https://opensearch-ipoid.discovery.wmnet:30443 invariably goes to eqiad. I believe that is due to the current DNS discovery settings, and I've prepared a patch to address this as well.
Change #1238741 had a related patch set uploaded (by Bking; author: Bking):
[operations/deployment-charts@master] opensearch-ipoid: Avoid 1G hosts
Change #1238741 merged by Bking:
[operations/deployment-charts@master] opensearch-ipoid: Avoid 1G hosts
Change #1238832 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] dse-k8s-ingress: Enable active-active
Icinga downtime and Alertmanager silence (ID=21e07a64-649b-4e9b-b3af-ae1793942269) set by bking@cumin2002 for 21 days, 0:00:00 on 9 host(s) and their services with reason: shut off 1Gbps hosts
dse-k8s-worker[1010-1013,1015-1019].eqiad.wmnet
We had to rollback a separate deploy due to the same latency issues. Due the way Kubernetes ingress works, requests were still being routed to 1Gbps hosts.
After depooling all 1Gbps hosts from Pybal, latency stabilized at ~35 ms. We still need to enable CODFW for OpenSearch iPoid, as discussed on Slack.
We'll update the ticket when we have more progress to share.
Change #1248087 had a related patch set uploaded (by Bking; author: Bking):
[operations/dns@master] dse-k8s: Enable active/active for dse-k8s clusters (2/2)
Change #1248087 abandoned by Bking:
[operations/dns@master] dse-k8s: Enable active/active for dse-k8s clusters (2/2)
Reason:
git hell, will resubmit
Change #1248097 had a related patch set uploaded (by Bking; author: Bking):
[operations/dns@master] dse-k8s: Enable active/active for dse-k8s clusters (2/2)
Change #1238441 merged by Bking:
[operations/dns@master] dse-k8s: Enable active/active for dse-k8s clusters (1/2)
Change #1238832 merged by Bking:
[operations/puppet@production] dse-k8s-ingress: Enable active-active