I'm getting "An error has occurred while searching: We could not complete your search due to a temporary problem. Please try again later.", which I assume is related.

I can't see any errors for Special:Search (they are somehow caught / hidden?) but do see some errors coming up from Special:MediaSearch

We could not complete your search due to a temporary problem. Please try again later.

from /srv/mediawiki/php-1.41.0-wmf.13/includes/api/ApiBase.php(1543)
#0 /srv/mediawiki/php-1.41.0-wmf.13/includes/api/ApiQuerySearch.php(147): ApiBase->dieStatus(Status)
#1 /srv/mediawiki/php-1.41.0-wmf.13/includes/api/ApiQuerySearch.php(72): ApiQuerySearch->run(ApiPageSet)
#2 /srv/mediawiki/php-1.41.0-wmf.13/includes/api/ApiPageSet.php(293): ApiQuerySearch->executeGenerator(ApiPageSet)
#3 /srv/mediawiki/php-1.41.0-wmf.13/includes/api/ApiPageSet.php(256): ApiPageSet->executeInternal(boolean)
#4 /srv/mediawiki/php-1.41.0-wmf.13/includes/api/ApiQuery.php(648): ApiPageSet->execute()
#5 /srv/mediawiki/php-1.41.0-wmf.13/includes/api/ApiMain.php(1913): ApiQuery->execute()
#6 /srv/mediawiki/php-1.41.0-wmf.13/includes/api/ApiMain.php(859): ApiMain->executeAction()
#7 /srv/mediawiki/php-1.41.0-wmf.13/extensions/MediaSearch/src/Special/SpecialMediaSearch.php(561): ApiMain->execute()
#8 /srv/mediawiki/php-1.41.0-wmf.13/extensions/MediaSearch/src/Special/SpecialMediaSearch.php(235): MediaWiki\Extension\MediaSearch\Special\SpecialMediaSearch->search(string, string, array, integer, string, string)
#9 /srv/mediawiki/php-1.41.0-wmf.13/includes/specialpage/SpecialPage.php(701): MediaWiki\Extension\MediaSearch\Special\SpecialMediaSearch->execute(NULL)
#10 /srv/mediawiki/php-1.41.0-wmf.13/includes/specialpage/SpecialPageFactory.php(1558): SpecialPage->run(NULL)
#11 /srv/mediawiki/php-1.41.0-wmf.13/includes/MediaWiki.php(328): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
#12 /srv/mediawiki/php-1.41.0-wmf.13/includes/MediaWiki.php(925): MediaWiki->performRequest()
#13 /srv/mediawiki/php-1.41.0-wmf.13/includes/MediaWiki.php(579): MediaWiki->main()
#14 /srv/mediawiki/php-1.41.0-wmf.13/index.php(50): MediaWiki->run()
#15 /srv/mediawiki/php-1.41.0-wmf.13/index.php(46): wfIndexMain()
#16 /srv/mediawiki/w/index.php(3): require(string)
#17 {main}

I was not able to replicate on special:mediasearch, only special:search. Is there a chance these are different bugs?

I don't know anything about search but from a quick look around I can not find any alarm being raised for whatever is happening. However I can see elevated log error on the backend (ElasticSearch) which started roughly on June 17th 9:00 (UTC) or 22 hours ago.

They apparently all fail on the wikidatawiki_content index and I see some errors such as:

All shards failed for phase: [query]
[Unknown analyzer [text_search]]; nested: IllegalArgumentException[Unknown analyzer [text_search]];
Caused by: java.lang.IllegalArgumentException: Unknown analyzer [text_search]

I am going to grab a coffee and pass some calls.

The elevated errors show up on Grafana dashboard for ElasticSearch Percentiles:

And on the dashboard for wikidatawiki_content index some metrics went down yesterday at 11:10 UTC.

I have texted some members of the Wikimedia Search time and they are going to have a look at it.

Change 930930 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/WikibaseCirrusSearch@master] token_count_router: infer the analyzer from the field

https://gerrit.wikimedia.org/r/930930

Change 930931 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@master] token_count_router: infer the analyzer from the field

https://gerrit.wikimedia.org/r/930931

Change 930909 had a related patch set uploaded (by Hashar; author: DCausse):

[mediawiki/extensions/WikibaseCirrusSearch@wmf/1.41.0-wmf.13] token_count_router: infer the analyzer from the field

https://gerrit.wikimedia.org/r/930909

Change 930910 had a related patch set uploaded (by Hashar; author: DCausse):

[mediawiki/extensions/CirrusSearch@wmf/1.41.0-wmf.13] token_count_router: infer the analyzer from the field

https://gerrit.wikimedia.org/r/930910

There are a lot of people affected by this based on how many people are asking about it in various places on-wiki adn on Telegram. I think this is worth "unbreak now".
Thanks for looking into it!

Change 930910 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@wmf/1.41.0-wmf.13] token_count_router: infer the analyzer from the field

https://gerrit.wikimedia.org/r/930910

Change 930909 merged by jenkins-bot:

[mediawiki/extensions/WikibaseCirrusSearch@wmf/1.41.0-wmf.13] token_count_router: infer the analyzer from the field

https://gerrit.wikimedia.org/r/930909

Mentioned in SAL (#wikimedia-operations) [2023-06-18T09:29:23Z] <hashar> mwdebug1001: scap pull of https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/930910 and https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseCirrusSearch/+/930909 # T339810

Mentioned in SAL (#wikimedia-operations) [2023-06-18T09:50:11Z] <hashar@deploy1002> Synchronized php-1.41.0-wmf.13/extensions/WikibaseCirrusSearch: T339810 - token_count_router: infer the analyzer from the field (duration: 14m 11s)

Mentioned in SAL (#wikimedia-operations) [2023-06-18T10:02:29Z] <hashar@deploy1002> Synchronized php-1.41.0-wmf.13/extensions/CirrusSearch: T339810 - token_count_router: infer the analyzer from the field (duration: 05m 50s)

❤

Change 930930 merged by jenkins-bot:

[mediawiki/extensions/WikibaseCirrusSearch@master] token_count_router: infer the analyzer from the field

https://gerrit.wikimedia.org/r/930930

Change 931271 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@wmf/1.41.0-wmf.13] token_count_router: infer the analyzer from the field (followup)

https://gerrit.wikimedia.org/r/931271

Change 930931 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] token_count_router: infer the analyzer from the field

https://gerrit.wikimedia.org/r/930931

Change 931271 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@wmf/1.41.0-wmf.13] token_count_router: infer the analyzer from the field (followup)

https://gerrit.wikimedia.org/r/931271

Incident report: https://wikitech.wikimedia.org/wiki/Incidents/2023-06-18_search_broken_on_wikidata_and_commons