@EBernhardson we realized a little late that @pfischer is OOO the next couple of weeks. Do you happen to know who might be his hypothesis delegate? It would be useful to have weekly reports continue there. :)

I'm not sure about the hypothesis delegate, but i can say the SLO's are DPE-SRE tasks, not search platform directly. Implementation will depend on their prioritization.

Also indexing latency already has an SLO: https://grafana.wikimedia.org/d/8xDerelVz/search-update-lag-slo?

I think it's worth taking a step back here when Peter returns, so we look at what the hypothesis is requesting and what's covered by existing SLOs and infrastructure, and what needs to be new.

The essential request is: let's agree on a reusable solution for indexing and searching across per-page structured data, generated from arbitrary data pipelines, with operational guarantees on indexing latency, serving availability and query latency.

From previous informal chats with the Search team, I'm guessing that the answer will be: "The correct path for satisfying those requirements is to push weighted tags to the CirrusSearch cluster, so let's add an indexing latency SLO for weighted tags, and availability and latency SLOs to that cluster."

But alternatively the answer may be "We need a new cluster to satisfy this requirement, with its own SLOs", or even "Weighted tags are the wrong answer, we need to design something new before we even think of SLOs." So let's talk about what's feasible here, and what other parameters we're missing to make a decision.

For what it's worth, the existing indexing latency SLO does look helpful here!

pfischer opened https://gitlab.wikimedia.org/repos/sre/slothslos/-/merge_requests/51

search: add weighted tags query latency SLO (T431145)

The query latency half of this hypothesis is now measurable, and I have a concrete SLO to
propose.

What changed. T434975 added two dimensions to
mediawiki_CirrusSearch_request_time_seconds: weighted_tags (yes/no) and pool (the
pool counter bucket). Both are merged and fully rolled out as of 2026-09-10, so the share
of request volume without the labels is 0. Weighted tag searches can now be selected
directly, which was the missing piece: previously they were reported as type="full_text"
and could not be told apart from ordinary full text search.

What the data says

External traffic only (mw-api-ext plus mw-web), 24h:

For comparison, over the same window and the same namespaces, all searches together are at
p95 254ms and the searches without weighted tags at p95 329ms. Weighted tag queries are
markedly cheaper than average search, being filter shaped and skipping most text scoring,
which is the reason the 200ms aspiration is met with room to spare today.

Proposed SLO

95% of searches filtering on weighted tags complete in under 250ms, measured on
external traffic, over a 30 day rolling window. Sloth spec in review:
https://gitlab.wikimedia.org/repos/sre/slothslos/-/merge_requests/51

Three things about that wording are worth agreeing on explicitly.

250ms, not 200ms. MediaWiki emits this latency into a histogram whose bucket edges are
shared fleet-wide by statsd-exporter: 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s,
5s, 10s, 30s, 60s. A threshold based SLI can only use an existing edge, and there is no
200ms edge. Committing to 200ms means first adding a CirrusSearch specific mapping to
charts/statsd-exporter/values.yaml in deployment-charts. Happy to do that if Derived Data
wants the round number, it just needs to land before the commitment is published.

Which traffic counts. Weighted tag queries by namespace, 24h averages:

Most weighted tag traffic is not user facing at all: mw-cron alone is roughly eleven
times the external volume, largely refreshLinkRecommendations. The SLI therefore covers
only mw-api-ext and mw-web, the two namespaces serving external traffic. mw-api-int
is excluded as internal API traffic, which costs about 16% of the events; the consequence
is that a request reaching MediaWiki through an internal service on a user's behalf is not
counted.

Why 95% and not 99%. Steady state is ~0.23% of requests over 250ms, so normal
operation consumes under 5% of a 95% error budget. Two bad days at a 25x degradation would
add ~0.33 percentage points to the 30 day ratio, about a tenth of that budget; against a
99% objective the same two days would eat half of it. Since the label has under a day of
history, I would rather start generous and tighten after a quarter of data than publish a
commitment we have to walk back. Note the window is 30 day rolling, so a bad weekend ages
out after 30 days rather than being cleared at the quarter boundary.

Caveats on the number

• Latency is measured by MediaWiki, so it includes pool counter wait and the network round trip, not just the OpenSearch took time.
• Cache hits never reach the histogram, so the denominator counts backend searches only.
• Availability is scopeable the same way now: mediawiki_CirrusSearch_backend_failures_total{weighted_tags="yes"} carries the same labels, if we want an availability SLI alongside latency.

Follow-ups

• A Grafana panel for the two new dimensions on the Elasticsearch Percentiles dashboard.
• Revisit the objective once a quarter of data has accumulated.
• Unrelated bug found while mapping the pool rules: Searcher::getPoolCounterType() keys on the syntax token deepcat, but DeepcatFeature::getFeatureName() records deepcategory and nothing records deepcat, so deep category searches never reach the expensive pool. I will file that separately.

pfischer updated https://gitlab.wikimedia.org/repos/sre/slothslos/-/merge_requests/51

search: add weighted tags query latency SLO (T431145)

I revised the Search SLO. The four query latency SLIs the page carried were defined on Graphite metrics that no longer exist, and the Prometheus metrics that replaced them measure backend time instead of time to render, so nothing was measuring them and they never made it into a sloth config. I left them out for now. They are still in the page history, so we can pick them up again once we agree on how to measure them.

What remains are the three SLIs that matter for Derived Data. Two of them are general anyway (availability, update lag), and the query latency one is scoped to weighted tag searches, following Derived Data's requirements.

@Gehel, please let me know if you are okay with this from a DPE SRE perspective.

@GGoncalves-WMF, all three objectives are at 95% for now, which is the usual starting point here. The 250 ms latency threshold follows the histogram buckets we have available; 200 ms would need a CirrusSearch specific statsd-exporter mapping first, and I would rather add that later than block on it now. Please read these numbers as the minimum we commit to, not as what to expect in practice: we currently measure 99.3% of weighted tag searches within 250 ms, 99.986% availability, and 98.5% for update lag. I am happy to tighten the targets once we have a few quarters of data, I would just rather avoid that overhead now.

Thanks Peter! Quoting the summary from the SLO page:

We publish the realistic targets: 95% for availability, for weighted tags query latency within 250 ms, and for update lag within 10 minutes. Each has substantial headroom, deliberately: Search Platform has no 24-hour on-call, so the error budget has to absorb an incident that begins outside working hours and is not mitigated until the next working day, without the SLO forcing a page overnight.

This looks good to me!

Closing this out. The SLOs are defined in slothslos under search/ and documented on SLO/Search. Recording here the reasoning and the measurements, which are point-in-time and do not belong on the page.

Measured, 30 days to 2026-09-29

Availability worst day 99.84%, worst hour 96.28% (over 90 days). The weighted tags figure is provisional: the label only reached full volume 14 days ago, and it has already drifted from 99.0% on the 16th to 98.4% today.

Why 95%

• 95% is the usual boundary for a new SLO; confirmed with core SRE. No dependent team asked for tighter, and the requesting side (Derived Data) accepted 95% together with the 250 ms bucket.
• The window is a fixed 90-day quarter, unshifted, so a quarter's budget is 4.5 days.
• Availability spends ~0.3% of that in steady state, i.e. effectively the whole budget is available for incidents. Latency and codfw update lag are the tight ones at roughly a third.
• Worth a look again after a full quarter, once weighted tags has real history.

Why a task and not a page

• Only core SRE has an on-call rotation and gets paged, and they do not know search infrastructure, so a page at 03:00 buys nothing actionable.
• DPE SRE has the knowledge but no rotation, and is not paged.
• At 95% over a fixed quarter a whole dark weekend costs about 60 of the 108 hours of budget. One fits; two do not. That is enough room to respond on the next working day.
• So: alerting by task only. Revisit if a tighter target is ever requested.

Why not cluster health as the SLI

• Over the same 30 days production-search-codfw was non-green 2.9% of the time and red 0.22% (~1.5 hours); eqiad was non-green 0.44% and never red.
• An SLI on health colour would therefore report ~97.1% for codfw while queries were actually succeeding at 99.983%. Yellow is routine shard recovery and still serves searches.
• Proxy-level measurement would sit closer to the clusters, but there is no envoy or LVS telemetry for the search upstream in the ops Prometheus to build on.

Sizes behind the caveats listed on the page

• Pool counter shedding is invisible to the SLI: 2.57 million rejections of 6.23 billion acquisitions over 30 days (0.041%, of which the Search pool is 951k). That is roughly 2.4x the failure rate we do measure. pool_counter_seconds_count has no weighted_tags label, so it cannot be scoped the way T434975 scoped the others.
• Partial results count as available: 444,709 log lines in 30 days, 419,330 of them (94%) regex, about 6.7% of all regex searches. Tracked in T438079: Searches that return partial results after a shard timeout are counted as successful and not measured.
• index_not_found_exception is ~236k errors a week, 96% of all non-user-error search failures, at 0.014% of traffic. Tracked in T438066: Roughly 236k index_not_found_exception errors per week on external CirrusSearch traffic.
• Update lag is the median of one batch, so half the events in it are slower and the tail is not visible.
• Searches answered from cache take successViaCache() and are in neither numerator nor denominator.

rzl merged https://gitlab.wikimedia.org/repos/sre/slothslos/-/merge_requests/51

Consolidate the Search SLOs into one manifest and adds an availability SLO (T431145)

✅ The SLO is implemented can be observed via Grafana.

https://grafana.wikimedia.org/d/slot-pilot-slo-detail/sloth-slo-detail?var-sli_window=5m&from=now%2FfQ&to=now%2FfQ&timezone=utc&var-service=search&var-revision=$__all&var-slo=$__all&var-min_burning_rate=1&var-hours_in_range=2209