Yea i think we can call this done, we had enough data to choose a model and the decision is reasonably documented.

cluster has been undeployed and PVC's deleted.

Fix works as expected:

Poking at mediawiki, what gets used is SpecialPageFactory::resolveAlias. It applies a couple extra rules against the alias list:

Cluster is up and looking well. The initial tests with a smaller working set are querying only frwiki. The larger working sets are all 4 indices we will run in the public test with request rate weighted by page views (results in ~88% enwiki). Note that the memory available to cache the working set is the reported memory minus 96G used by opensearch directly. Tests were run for ~30k requests each after a warmup period.

Cluster is initialized and is currently loading data.

We could turn off interwiki search, which saves a good number of commonswiki_file searches (a very large index)

i wonder if we should keep deepcat as it's own pool, or perhaps fold it into the expensive-full-text pool that we generalized regex into?

I tested a few deploys today, but didn't quite get it going. I'm currently trying to deploy with 16nodes at 16gb each. 4 have been allocated, it looks like the next set of nodes is being blocked due to not having enough memory available. Will check-in with SRE.

This is now published as docker-registry.wikimedia.org/repos/search-platform/cirrussearch-opensearch-image:v3.8.0-0. Still need to verify cindy passes using this image.

Looks like there are some docs hinting at a way forward: https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html

This is stalled, the current process to build .deb's in gitlab CI is busted. Looks related to T437069 and T437829. We need new weekly versions of the wmf-debci-* images to be released.

Posted: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Curating_redirects_via_search

I've drafted a thread for WP:VPT to collect use cases for more redirect specific use cases. We should still think about how to collect these usecases outside enwiki though. Intending to post this on monday, I didn't want to put it up on friday before i disappear for the weekend.

All plugins have now been released for 3.8.0. Updating the plugins .deb and the related opensearch docker image are in-progress and will be completed in T435824.

mjolnir ran it's weekly job to success.

In T204089#12264167, @Nardog wrote:

So is the exclusion still not implemented? Or has the default behavior changed so that a search doesn't cover redirects unless prefixed with withredirects:?

In T204089#12264500, @STei-WMF wrote:

@EBernhardson can we add more details?:

Who is the target audience for this entry? – I believe this would be readers and editors searching for pages

I added some docs, at least as a starting point, to Help:CirrusSearch. I suspect this will need to be further refined a bit, it feels unclear but I'm a bit too close to the project to pin down what exactly. Will seek some review.

Indeed onlyredirects: (search only against redirects) and withredirects: (combined search against all known pages) have to be a query prefix, this was done for a few reasons that i will try to properly document.

This looks to be working as expected. Using english wikipedia as an example, 12M redirects in content namespaces and 15M redirects across all namespaces are now searchable using typical keywords like insource://, intitle://, etc.

To create the cirrus-specific opensearch 3.8 image we will also need a base 3.8 image. will do both in this ticket.

This is merged and deployed, we will want to wait for the weekly run to happen to make sure there is no funny business in the migration. Not expecting any problems, but intending to come back around to this on monday after the run to mark as finished.

I can add another sentence selector to the evaluation set, in the existing evaluation we did evaluate a neural sentence selector (jina-reranker and qwen3-reranker) which performed quite well but was far too compute heavy (~30 passages/sec on gpu). It seems plausible a cheaper sentence selector could get similar results.

mjolnir looks to have built appropriately: https://gitlab.wikimedia.org/repos/search-platform/mjolnir/-/packages/2431

@pfischer you are probably the closest available for looking over CirrusSearch code

Patch written and tested: F99026686

On further review, this problem fixed itself without intervention. From grafan the elasticsearch_indices_age_days{index="*_titlesuggest"} metric, which records the age of the oldest titlesuggest index, increased from 8-14 to 8-18, meaning at least one index wasn't reindexing. Today though, on the 18th, everything has reset back to age < 1 day without any specific intervention. Whatever was wrong got fixed elsewhere.

Started looking into this, not clear yet:

Checked for apifeatureusage2001 logs:

I've spent some time working this out and have a preliminary report which finds mdeberta is likely our best choice moving forward. Needs review.

In my testing i was seeing ~125 passages/s. It looks like you are seeing 40req * 3 passages = 120 passages/s, or 32 req * 5 passage = 160 passages/s, so very similar numbers.

In T434008#12189137, @EBernhardson wrote:

I worked through this with claude and managed to build a local reproduction of the issue along with a potential resolution. Our initial theory was cluster state size, but it looks that that is not that case. Full findings stored P95921, the summary is:

The master election storms are **not** primarily caused by cluster state size. They
are caused by a per-node-join CPU cost that scales with the number of indices
carrying `auto_expand_replicas`, via a code path that **does not exist in
OpenSearch 1.3.x**. State size makes it worse, but is not the trigger.

I'm now working on a fix in CirrusSearch, we don't actually take advantage of auto_expand_replicas, we were simply using it to have a single code path for both us and small clusters outside WMF. We actually set these to constant values anyways (auto_expand_replicas: 2-2).

I worked through this with claude and managed to build a local reproduction of the issue along with a potential resolution. Our initial theory was cluster state size, but it looks that that is not that case. Full findings stored P95921, the summary is:

Merged and released, but we'll want to wait for monday now to deploy it.

reindexes are completed in eqiad and codfw, but cloudelastic is struggling to complete the tasks. This most likely has more to do with the recent upgrade from opensearch 1.x to 2.x, than anything specific to this change, but will delay things by a little bit while we figure out why it's not happy.