@matmarex encountered similar issues and suggested a change in how we handle wprov when arriving on the landing page (see https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/1267282). Haven't found a way to reproduce this but did not look closely on what happens with the RelatedArticles extension. I'm all for more standardisation on this.

In T418525#11833996, @atsuko wrote:

Also it would be good to update search-platform/cirrus-streaming-updater/.pipeline/blubber.yaml, but it needs maven build to make it work. cc @dcausse

Here is proposed diff https://gitlab.wikimedia.org/-/snippets/290

In T422928#11833564, @Ottomata wrote:

Thanks @JMeybohm and @dcausse. I'd like to try this!

So, IIUC we should:

• set x-envoy-max-retries: 0
• set x-envoy-upstream-rq-timeout-ms to our client timeout - 100ms (ENVOY_OVERHEAD_MS) ?
• set x-forwarded-for: 127.0.0.1

Is that correct?

Is there a way to the effect? Somewhere and grafana that shows envoy retries for our service? It'd be nice to see the effect explicitly rather than just hoping. :)

For the retries I would expect to only see "limit_exceeded" in the upstream request retry panel from the envoy telemetry, regarding timeouts I'm not so sure but possibly you should rarely see requests timeouts from your app code but only 504 upstream response timeout?

In T422928#11828227, @JMeybohm wrote:

From what I recall there are certain limitations on the headers that can currently be used to reconfigure envoy's behavior from client side since T354853: Service mesh envoy does not treat incoming connections as local. As a workaround an additional service-mesh listener with the expected retry policy could be created/used.

We sometimes use search backend query logs for this, the table is event.mediawiki_cirrussearch_request.
See P90838 for a quick example. A quick extraction shows that 0.64% (1,244,901 out of 193,639,885) of the completion requests are using strictlanguage for the first half of April 2026.

Applied a quick workaround in WikibaseCirrusSearch test case, briefly looked over WikibaseLexeme but was unsure about what to do. I think the current design of EntityHandlerTestCase is not flexible enough and assumes that all subclasses have to run all the tests, it seems to me that an approach around composition with traits might be more appropriate but it sounds like a non trivial refactoring.

In T423327#11822826, @RKemper wrote:

Proper fix:

OpenSearch supports configuring path.plugins in opensearch.yml. If we set it to something like /usr/share/opensearch/wmf-plugins and rebuild wmf-opensearch-search-plugins to deploy there, we sidestep the dpkg conflict entirely. opensearch ignores the upstream-bundled plugins in the default path and only loads from our custom path. No strip/reinstall dance, no dpkg conflicts, survives upgrades cleanly.

This seems like a nice trick to prevent opensearch from loading its default plugins.

In T418130#11817083, @TJones wrote:

Things generally look good—and I'm really excited about the second-try searches I see!—but there's one thing that's a little off, though maybe you've accounted for it.

Entries that have position "10" mean that they rolled over to fulltext search, right? (Because the suggestions are 0-9.)

These include all the ones from kawiktionary, one from ruwiki, and two from kawiki. The page titles associated seem to be at least kinda close, but, for example, if you search for "катет" on kawiktionary, you get 10 suggestions in Cyrillic, so there shouldn't be any second-try suggestions shown.

indeed, I checked the code and the last line of the dropdown "search for page containing $query" is tracked as a click and was wrongly assigned to the last results of the backend logs, excluded those to count them as "srp_clicks".
Results are roughly similar (between 1% and 3% of the interactions are due to dwim on wikipedias):

In T417694#11815172, @MNeisler wrote:

@xcollazo

searchsatisfaction — ~2.3M files, 6,426 GB total, actively written, data goes back to 2021. Retention was explicitly requested in T274607 for analytics use. A long-term retention policy may be worth establishing

I no longer work regularly with this dataset. From the product analytics side, I believe it would be fine to delete some of the older data; however, I'd recommend checking with the Search Platform team to confirm. Maybe @EBernhardson or @dcausse?

@TJones I extracted a sample a query clicks where the clicked result comes from DWIM, if you have a moment could you check the data?
It's at /user/dcausse/T418130-dwim-analysis/sample.csv and can be extracted locally from stat machine using hdfs dfs -text /user/dcausse/T418130-dwim-analysis/sample.csv/part* > dwim_sample_query_clicks and let me know if you spot anything weird (mainly trying to make sure my data extraction is not incorrect).

Looked briefly into the data as we should be able to join frontend and backend logs and results over 1.5day are quite promising:

In T422511#11807568, @gmodena wrote:

Unless flink is doing something special during dynamic partition discovery it might use its default offset reset strategy? If yes and they use latest offsets they might lose events the time the app is restarted or partitions discovered:

• IIRC wdqs uses latest and may lose events for at most 5min (default partition discovery interval)

Urgh. You are right.

FLIP-288 suggests that later discovered partitions will reset from earliest. But this has been implemented in kafka connector >= 4.0.0 (we are still on 3.x.x).

@Trokhymovych please find the query result pairs in /user/dcausse/semantic_search/T419397/query_result_pairs/ (file names should be self-explanatory).
My problem with multilingual-e5-large-instruct was due to its small context size, around 300k passages were affected, I reran those allowing the model to ignore extra tokens.

In T422511#11805626, @Ottomata wrote:

But, perhaps it would be better if applications didn't rely on canaries for watermark advancement? They are more intended for a pipeline health check than a pipeline 'liveness probe' :)

Flink apps generally set source idleness and don't expect canary events but unfortunately they can't avoid these events to still wake-up idle sources (unless there are ways to apply a filter very early at the source level?).
I actually don't know if this is a problem to have partitions properly idle and a single one being woken up at regular intervals, it might just be OK but sounded a bit odd to have only one partition receiving this traffic.

In T422511#11800369, @Ahoelzl wrote:

@dcausse @gmodena @gkyziridis please provide your respective team input.

@Trokhymovych I think I'll have the data by the end of day.
I have the embeddings extracted and indexed for

• pplx-embed-context-v1-0.6b: 53M passage in 36h (1024 cores with beefier workers: 64*4G+16G, 16cores)
• embbedings-jinaai-embeddings-v5-nano: 53M passages in 19h (1024 cores, workers: 64*4G+8G, 16 cores)

Impact: the semantic search API returned results from a very small subset (~10%) of the articles during these periods:

• ptwiki: 2026-05-05T14:00:00 to 2026-05-08T08:41:00
• enwiki: 2026-05-05T15:30:00 to 2026-05-08T13:56:00
• frwiki: 2026-05-05T15:15:00 to 2026-05-08T16:00:00

Looking at idwiki I see a huge difference in the number of passages extracted:

20260329	3750496
20260405	724490

I understand the issue as follow:

• You search for "pet" hoping to select Pet (Q39201)
• You see a wide variety of matches starting with pet but not what you expect (Pet Q39201)
• You add a space and search for "pet " hoping for Q39201 to be preferred

@Ahoelzl yes I can confirm the run scheduled on April 2nd did run properly.

The trailing spaces are not trimmed by MW but because the completion algorithm is tokenizing the query and the titles which will cause spaces to be ignored most of the time.
We can attempt to change the behavior but I'd suggest to run a quick test to see possible undesirable side-effects.
There is also an an issue with wikibase prefix search (which is not using the same algorithm) where trailing spaces are supposed to influence scoring but this got disabled by some custom scoring profiles.
We can fix custom profiles (en, de, es fr) to let trailing spaces influence the ranking but I think this is different from what is requested in T417648.
In T417648 I understand that the trailing space should be used as a signal to tell the search engine to look for "exact" matches on the trimmed search string. The wikibase prefix search should already have a ranking signal on an "exact" label/alias match from the trimmed search string but I suspect the different weights are not doing what we expect in this case.

In T419397#11778612, @Trokhymovych wrote:

As for your question, I think the proposed approach should work. That said, I would recommend including the title and section context (e.g., title, parent sections, and section name) in all paragraphs, not just the first one.

jina-embeddings-v5-text-small is done: 53M passages in 51.3h with 1000cores

In T368987#11776496, @APizzata-WMF wrote:

The task finished running during the night, @dcausse do the numbers look good now?

@Trokhymovych sure, please find those in hdfs:///user/dcausse/T419409-miracl/query_result_pairs with names pure_knn_10_top_10_per_article_miracl_$lang.json.
I collected the top-10 passages per article such that in the worst case (if best matches all belong to the same page you should have them). If you flatten the passage arrays re-ordering by score you should be able to infer the top-10 passages per query. Please let me know if you spot issues and I can try re-shape my query and output format to ignore the per-article breakdown if problematic.

In T368987#11771834, @APizzata-WMF wrote:

All looks good to me, @dcausse if you agree I will run the task publish_page_change_weighted_tags.

Awesome, thanks! yes please feel to re-run this task

Quick update

• multilingual-e5-large-instruct is done: 53M passages in 29h with 1000cores
• jina-embeddings-v5-text-small is running (started yesterday and is 40% done)

In T368987#11763867, @APizzata-WMF wrote:

I have created and validated all my update commands, will shortly run them all and past here all the results with the validations. As a next step we can just rerun the image_suggestions_weekly dag correct?

I think one of the cause of this confusion is the current contract between ALIS/SLIS and search, for historical reasons (first non-search pipeline to push data to search indices) it used search internal formats which are very fragile (magic words like __DELETE_GROUPING__ and tag/score internal string encoding). I filed T414099 to try to address some of this a update the ALIS/SLIS to use newer contract based on an event schema which is what newer pipelines are using (i.e. revise tone recommendations).

In T368987#11763303, @APizzata-WMF wrote:

Hey @dcausse I must have misunderstood and thought the exists| part could be removed. I can update the output of the tables to show the correct form and fix the code to show in the correct form from next run. Does this sound good to you?

@APizzata-WMF sorry I did not spot this earlier but the values ["970"] is not correct, it unfortunately pushed the string 970 as the tag value instead of the score, the tag value must remain exists with the score appended after a |: the full string should look like ["exists|970"].
Is this possible to re-run the pipeline with such fix applied, I'll re-ship the tags right after.

Update:

• jina-embeddings-v5-text-nano is based on eurobert and llama.cpp got it supported just recently and spark-nlp does not support it yet, tried to rebuild spark-nlp with it but faced a blocker, skipping for now but an alternative might be not using spark-nlp and fallback to hf sentence transformers (this requires some adaptation to the current pipeline)
• pplx-embed-v1-0.6b might be a bit complicated esp. if we want to take benefit from the context model for which I need to adapt the pipeline to emit meaningful batches, I suspect I'll have some questions about how to build the batch to keep a meaning full context
• started to extract embeddings with multilingual-e5-large-instruct using a quantized (Q8) version I built with llama.cpp tooling and uploaded to /user/analytics-search/spark-nlp/models/multilingual_e5_large_instruct_Q8_0_gguf

@MGerlach @Trokhymovych query result pairs should be available in hdfs:///user/dcausse/T419409-miracl/query_result_pairs, files are named pure_knn_10_miracl_$lang.json.

Unfortunately the job failed a couple times last week and embeddings extraction just finished last night, recording some timings (1000cores with llama and qwen3-0.6B-Q8):

• de: 15.8M, 19h
• en: 32.8M, 28h
• fr: 14.6M 13h
• id: 1.4M 2h
• es: 10.3M 11h

Started the job at https://yarn.wikimedia.org/cluster/app/application_1773845446826_8538 the dataset is quite big and probably will take quite some time to complete (hopefully finished early next week).

@Trokhymovych it's working well now, no problem, thanks for the data!

@Trokhymovych thanks! I don't have access to these folders, could update the perms or possibly upload them to /user/dcausse/T419409-miracl if you don't want to open the perms on your user folder?

Tentatively raising prio because it's failing CI on all CirrusSearch patches too and I suspect some other repos as well.

I seem to get failures on ve-cite. I got a failures 3 times consecutively on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/1252597 (quibble-vendor-mysql-php83-selenium):

• https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83-selenium/31154/console
• https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83-selenium/31268/console
• https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83-selenium/31284/console

Embeddings for dewiki, enwiki, eswiki, frwiki, itwiki, nlwiki, ptwiki and idwiki (main namespace only) are extracted weekly on Sundays and available in the table discovery.wiki_content_embeddings (partitions: snapshot, wiki, model='qwen3_0.6B_Q8')

Please find query-result pairs at: hdfs:///user/dcausse/semantic_search/T417242/. The folder contains a set of outputs:

• pure_knn_10.json: the top 10 of the bare vector search
• rerank_at_3_no_context.json: same as above but the with top-3 re-ranked using the bare passage
• rerank_at_10_no_context.json: same as above but the top-10 are re-ranked
• rerank_at_3_full_context.json: top-3 re-ranked using the passage with additional context (title, parent sections and section)
• rerank_at_10_full_context.json: same as above but the top-10 are re-ranked
• rerank_at_3_with_lead.json: top-3 re-ranked using the passage with additional context (title, parent sections and section and the lead paragraph of the page if different from the best passage)
• rerank_at_10_with_lead.json: same as above but the top-10 are re-ranked

@TJones thanks for the feedback and suggestions!
We have a repo at https://gitlab.wikimedia.org/repos/search-platform/notebooks but even if some notebooks are re-usable you sometimes have to duplicate and adapt it to run your analysis. The double period issue is I think me misusing some of the templates created by Erik.
I tried to update the last report to include more detailed info (per wiki graphs) but unfortunately I waited too long and some of the data is already gone thanks to our retention policy...
A/A test is a nice idea, haven't had the chance to do it but there's possibly a way to do it from the data we have? Some wikis have a real low volume (from what I remember bi, gv and igk had fewer than 100 observations in a week) and where we might want to double check https://foundation.wikimedia.org/wiki/Legal:Data_publication_guidelines and possibly start using thresholds rather than actual numbers.

@RoyZuo thanks for raising this. Indeed the ranking strategy for commons is quite suboptimal and the fuzzy matches tend to hide interesting matches from other namespaces. This would require a change in how the suggestions coming from multiple namespaces are blended together.
While we ponder how to prioritize and solve this you can apply a workaround by selecting either:

• Redirect mode
• Classic prefix search

from your Search User preferences.
This workaround might help in some cases but not for some others (hopefully rarer) where an "exact" category match is still a widely used prefix.

Currently the indexed search tags have a hard-coded value named exists but it could be something else, (if deemed useful at search time) tags related to image recommendations could be more granular by including the kind as separate tags.
At query time this would become (suggested syntax): hasrecommendation:image=istype-depicts.

• hasrecommendation:image>0.8 -hasrecommendation:image=istype-depicts could filter recommendations with a score above 0.8 and without a istype-depicts kind.

In T413969#11523393, @EBernhardson wrote:

This will probably be a couple parts:

Dumps are available as text files (raw ndjson) under /wmf/data/discovery/wikimedia_enterprise/structured_content_snapshots/snapshot=$YYYYMMDD/project=${WIKI}_namespace_0 and will be updated weekly on Sundays.

@gmodena the list of dags seem correct to me, there'll be some parts of drop_old_data_daily.py that might be moved over as well (cleanups of rdf data from import_ttl and query analytics).

In T414066#11511114, @awight wrote:

The full HTML snapshots are "Updated twice-monthly (on the 2nd and 21st)" (1) so I'm curious to know whether Structured Contents follows that same cadence.