In T411169#11611012, @STei-WMF wrote:

@EBernhardson thank you. Should it go out in Monday 16th's Tech News? (The wording suggests so, but I want to doublecheck)

I think the existing lead section should be short enough, something like:

Not entirely sure, but this might be reasonable to include in tech news.

In T401590#11570234, @KSiebert wrote:

Is the related wish resolved?

In T414763#11547081, @Pelikana wrote:

Today I found a bug in cases where the category name has an "&".
Example: deepcat:"Pellerin & Cie" Try a deepcat search in deepcat:"Pellerin & Cie" > results in this search: deepcategory:"Pellerin_ No matches because of the wrong folder name. The folder name is truncated at the "&".
I know there must be a patch for this since my branched button that automates " " -deepcat:" " (such as "Pellerin & Cie" -deepcat:"Pellerin_&_Cie") has this a patch been applied already for a longer time and works well at this point time. The fork is hosted and patched by User Samwilson.
I hope the regular deepcat function can be patched for this issue without damage to the opposite -deepcat searches.

This is now running again. It has completed the generation stage and is currently creating fresh glent indices in opensearch. At a high level this looks like, but i don't have old enough data to verify 100%, it was caused by a change in meta.dt timestamps to now include millisecond precision, likely as part of T376026.

In T414763#11542114, @Prototyperspective wrote:

Indeed this seems solved. However, the following still doesn't work and I meant to create a separate issue about this just before this bug here appeared: is this a separate problem or the same?:

• -deepcategory:"Maps of the world by language" deepcategory:"2020s maps of the world" -deepcategory:"Wikimania Map of the world" shows 230 results with no warning at the top about incomplete results
• the same after appending -deepcategory:"SVG maps of the world by language" -deepcategory:"English-language SVG maps" -deepcategory:"Our World in Data food and agriculture maps of the world" shows only 57 results (also without warning) but all of the added cats are subcats of the cats in the prior scan

(this example is what's used to populate 2020s_maps_of_the_world_in_unidentified_languages which is how at least / starting with the most relevant world maps are categorized by language to e.g. better enable translations and hopefully eventually better search results that doesn't show maps in some niche language I can't read at the top when that's not in my configured language(s))

Example query from description now works as expected

In T413969#11524363, @dcausse wrote:

If using the DefaultSearchQueryDispatchService a new "semantic_search" profile context will have to be created and this one will dictate what rescore profile to use and could be forced to "empty".
If a user explicitly selects a rescore profile (not using engine_autoselect for ftqiprofile) I think we could re-route to classic fulltext (the SearchQuery class keeps the list of forced profile and this can be inspected to make this decision).
Haven't looked closely but this could be an indication that using ftqbprofile to route to the knn query builder might not play well as it gives the false impression that you can assemble query bits freely. So perhaps using another criteria (custom cirrus param for now?) to trigger the knn search could be easier, the query router could cancel the knn query if some profiles have been selected explicitly (ftqiprofile/ftqbprofile != engine_autoselect).

In T413969#11525371, @bking wrote:

Per Erik's comment above:

elasticsearch-percentiles is often the first stop when receiving latency alerts to understand what might be going on.
elasticsearch-per-node-percentiles would be convenient for understanding latency profiles of the semantic search shard requests.

This important metrics are sourced from our custom exporter, which is not currently available in OpenSearch on K8s. (See T414345 for a further discussion). @EBernhardson / @pfischer , would you consider these metrics a hard requirement for the project? If so, we might have to:

A. (SRE) Modify the helm chart to run our custom exporter as a sidecar
B. (Search Platform) rewrite our custom exporter as an OpenSearch plugin.

Let us know if this is a hard requirement and if so, level of effort for implementing the exporter in Java as an OpenSearch plugin. SRE will also look into making a docker image for the exporter and how to integrate it into the chart if these metrics are indeed required.

@dcausse @CDanis @bking @RKemper The above plan is my first draft of how we go from where we are today, to having semantic search available for testing in production. Please review.

This will probably be a couple parts:

Documentation has been placed: https://www.mediawiki.org/wiki/Help:CirrusSearch/Debug

The latency buckets might be a little harder to upstream, but still possible. The latency buckets require special data collection implemented in our opensearch-extra plugin. Totally possible, but means upstreaming to both the main opensearch project and then the exporters once the data is available. It might require a slight re-architecture of the latency collection depending on what upstream thinks of the current implementation. We did propose this to ElasticSearch back in the day and they gave advise for implementing the latency bucket collectoin, but weren't interested in upstreaming.

This looks to now support ElasticSearch in REL1_45

Turns out the problem with the content was a cache on my end, issuing a force-refresh loaded the new content. The link is now pointing to the new content.

Workin on the documentation at P86859. This is still very preliminary, but once it's all pinned down better it will end up somewhere on mediawiki.org

In T411169#11413479, @ChildrenWillListen wrote:

As the person who filed T410602, it would have been nice to have an API taking in a search query and returning a set of results, as well as explaining why these results were chosen (e.g title, related, displaytitle, defaultsort, etc.) and their ranking (pageviews, etc.) Possibly unrelated, but a Wikitech/Mediawiki page containing a step-by-step guide on how the Search system works (as well as the function names/files involved in each step) would have been helpful as well, since I found the codebase to be labrynthine.

Deprecation doc has been placed, this should be complete.

Patches shipped, 20260104 dump was rerun and looks reasonable. I imported the simplewiki dump into a local instance and it loaded without issues.

I went through today to verify if everything is ready to go:

Something can be done to improve some of the use cases around redirects, but we would need to narrow in on things that can be done. The fundamental limitation here is that in the search data model redirects are not their own pages. The only metadata about redirects stored in the search index is the namespace and title, and that is attached to the page that is redirected to. Additionally search results are always at the granularity of the indexed documents. This means if two redirects to the same page match that can not be represented in the output as two matches. It will always be a match against the document that was redirected to, with a scoring bump for matching twice.

In T366248#11492787, @BTullis wrote:

Should we start by disabling the legacy cirrussearch dumps in the Airflow UI?
https://airflow-test-k8s.wikimedia.org/dags/mediawiki_cirrussearch_dump/grid

If nothing falls over and nobody complains after a couple of weeks, then we can remove the code from Airflow-DAGs.

After pondering this one, we think it's best to leave the timestamps as-is. The only "correct" timestamp is the publicly facing dump, changing and re-aligning all the internal timestamps is a good bit of work that seems helpful but ultimately unnecessary.

I think thiemo has got it pinned down, this is "working as intended". It would be convenient if the mediawiki api had a more intuitive method to align limits across the api call, but that's not currently a thing.

The code fix itself ended up being pretty straight forward. We might use this opportunity to re-run the most recent dump, learn a bit more about how replacing an already published dump would work.

In T408154#11408732, @TheDJ wrote:

It seems worthwhile to let the test continue running for a second week

@EBernhardson it's now two weeks later. Just in case this dropped off the radar by accident.

A temporary 3-node cluster has been stood up in T410681. This is running opensearch 3.3.2 and is accessible from the analytics network (stat machines, hadoop, etc.).

The initial parts of this are complete. The three node cluster has been stood up, and it is accessible from the analytics networks. As a test instance we didn't setup a dns service or tls, but i suspect that is acceptable. The cluster can be accessed on relforge1008, relforge1009, and relforge1010 on port 9200.

Hmm, that does seem likely. If we add a &cirrusDumpQuery to one of the searches we can see it has timeout: 15s, when indeed regex should get a longer timeout. Not sure yet what changed to cause that.

In T403212#11393825, @doctaxon wrote:

@EBernhardson This seems to throw timeout reports, see this dewiki discussion: https://de.wikipedia.org/wiki/Wikipedia:Fragen_zur_Wikipedia#Zeit%C3%BCberschreitung_bei_Insource-Suche

In T408133#11370867, @phuedx wrote:

In T408133#11366050, @pfischer wrote:

Regarding CirrusSearch: Do you already have particular open questions/obstacles related to the A/B test scenarios covered in the backend?

• Session starting by following a link to blank Special:Search
• Session starting by following a link to Special:Search with a query
• Session starting at Special:Search with a 'go'

Thanks @pfischer. A couple of questions spring to mind:

1. Is Scenario 1 treated the same as the other scenarios? In my mental model, a user searching or interacting with the search autocomplete starts a session. Is that correct?

Shouldn't have anything to do with time between requests, Deep category search SPARQL query failed mostly means one of the backend services had an error and the frontend should retry. It looks like instead the frontend is treating an error as the end of results.

In T403212#11377377, @A_smart_kitten wrote:

@EBernhardson apologies for bumping this, but do you think it might be worth me filing a follow-up feature request for this, given the similar usage/support in other regex flavours/libraries?

In T403212#11372531, @Bewfip wrote:

I noticed that surrogate pairs can't be used inside a character class. For example, both insource:/😂/ and insource:/[😂]/ work, but insource:/[\uD83D\uDE02]/ returns nothing. Using it in character classes can be handy when searching for a range of Unicode code points. Shall I file a new ticket for this?

Test has been out for a week, ran the notebook but results are curious. In particular we are seeing a significant change in ZRR, even though the test treatment does not change the retrieval function. This suggests we could have some unbalanced effects in the bucketing. It seems worthwhile to let the test continue running for a second week and run the notebook against the second week to verify the results.

Checking the last 2 weeks of logged Elastica HttpException there were 5 total exceptions, fairly low volume.

In the communication we went with promising dumps through november, shutting off sometime in december:

Curl error 52 is "Empty reply from server."

In T366248#11343661, @brouberol wrote:

@EBernhardson if you're happy with these new dumps, do you still want the "old" cirrussearch dumps to run on Airflow?

There is still a remaining problem with the query-time highlighter. The elasticsearch highlighter doesn't support the lucene_anchored flavor so we would need to always request lucene on elasticsearch. We are really trying to avoid extra query-time round trips to the server to determine the version information though, still pondering appropriate solution. It might be the only reasonable way is to remove anchored trigram support from REL1_45

Talked this over with @dcausse. We agreed we should continue to support Elasticsearch in REL1_45. We are adding a workaround for this bug with the regex support, and will add warnings that will be displayed when running scripts that manage search indexes whenever the indexes exist on an elasticsearch instance. The intention is to only support OpenSearch in REL1_46 and beyond.

With puppet deployed should expect to see these arrive at https://dumps.wikimedia.org/other/cirrus_search_index/ after 05:00 UTC tomorrow.

First run of the updated dag completed, dumps were formatted and moved to the exports path in hdfs. Reviewing the output it all looks reasonable and as expected. Next up is to enable the public sync via the puppet patch.