"We could not complete your search due to a temporary problem." searching words on Minangkabau Wiktionary
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	Harditaher
	Dec 15 2019, 9:09 AM

Description

I found "We could not complete your search due to a temporary problem. Please try again later." in searching words on min.wiktionary.org

Steps to Reproduce:

Go to https://min.wiktionary.org
at search box, type "namo" (this is only as an example of many words)

Actual Results:

"namo" is not yet a page. but there is a notification " Kasalahan tajadi wakatu mancari: We could not complete your search due to a temporary problem. Please try again later."

Expected Results:
There is no red notification and I expect that there are some articles contain word "namo"

Related Objects

Mentioned In: T295192: missing search index on checkuserwiki
T254331: Suspicious mismatch between psi and omega elastic cluster
T196740: Rename Wikimedia project tags in Phab (#Wikisource; #Wiktionary) to clarify that they are not about language specific instances?
Mentioned Here: T27682: Searching is broken in newly created projects

Event Timeline

Harditaher created this task.Dec 15 2019, 9:09 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 15 2019, 9:09 AM

This is similar to T27682, from nine years ago. A lot has changed since then in the search infrastructure, but search is still sometimes broken in new wikis. Is this something that can be automated? Or at least documented better in https://wikitech.wikimedia.org/wiki/Add_a_wiki ?

Thanks!

(I'm not sure what is the right tag for search bugs, please correct as needed.)

Removing All-and-every-Wiktionary tag; please see its description: "Please do not report language specific tasks under this project (e.g. for en.wiktionary)."

In T240778#5742268, @Amire80 wrote:

This is similar to T27682, from nine years ago. A lot has changed since then in the search infrastructure, but search is still sometimes broken in new wikis. Is this something that can be automated? Or at least documented better in https://wikitech.wikimedia.org/wiki/Add_a_wiki ?

Thanks!

(I'm not sure what is the right tag for search bugs, please correct as needed.)

Lucerne then, ElasticSearch now

Be interesting how similar the actual issue is

Looking at the CirrusSearch log on mwlog1001

2019-12-15 12:04:26 [XfYhSgpAICAAAKoT7DgAAABJ] mw1330 minwiktionary 1.35.0-wmf.10 CirrusSearch WARNING: Search backend error during near_match search for 'namo' after 7: index_not_found_exception: no such index {"queryType":"near_match","tookMs":7,"query":"namo","limit":50,"syntax":["near_match"],"index":"minwiktionary_content","hitsOffset":0,"suggestionRequested":false,"error_message":"index_not_found_exception: no such index"} 
2019-12-15 12:04:26 [XfYhSgpAICAAAKoT7DgAAABJ] mw1330 minwiktionary 1.35.0-wmf.10 CirrusSearch WARNING: Search backend error during full_text search for 'namo' after 3: index_not_found_exception: no such index {"queryType":"full_text","tookMs":3,"query":"namo","limit":21,"syntax":["full_text","full_text_simple_match"],"index":"minwiktionary_content","hitsOffset":0,"suggestionRequested":true,"error_message":"index_not_found_exception: no such index"} 
2019-12-15 12:04:27 [XfYhSgpAAEAAAJsLlY0AAAAK] mw1269 minwiktionary 1.35.0-wmf.10 CirrusSearch WARNING: Search backend error during near_match search for 'namo' after 7: index_not_found_exception: no such index {"queryType":"near_match","tookMs":7,"query":"namo","limit":50,"syntax":["near_match"],"index":"minwiktionary_content","hitsOffset":0,"suggestionRequested":false,"error_message":"index_not_found_exception: no such index"} 
2019-12-15 12:04:27 [XfYhSgpAAEAAAJsLlY0AAAAK] mw1269 minwiktionary 1.35.0-wmf.10 CirrusSearch WARNING: Search backend error during full_text search for 'namo' after 2: index_not_found_exception: no such index {"queryType":"full_text","tookMs":2,"query":"namo","limit":21,"syntax":["full_text","full_text_simple_match"],"index":"minwiktionary_content","hitsOffset":0,"suggestionRequested":true,"error_message":"index_not_found_exception: no such index"} 
2019-12-15 12:06:28 [XfYhxApAICIAAC@e-38AAACY] mw1332 minwiki 1.35.0-wmf.10 CirrusSearch WARNING: Search backend error during full_text search for 'Betina' after 4: index_not_found_exception: no such index {"queryType":"full_text","tookMs":4,"query":"Betina","limit":1,"syntax":["full_text","full_text_simple_match"],"index":"omega:minwiktionary_content","hitsOffset":0,"suggestionRequested":false,"error_message":"index_not_found_exception: no such index"} 
2019-12-15 12:06:29 [XfYhxApAME4AABEXVnYAAADG] mw1243 minwiki 1.35.0-wmf.10 CirrusSearch WARNING: Search backend error during full_text search for 'Betina' after 4: index_not_found_exception: no such index {"queryType":"full_text","tookMs":4,"query":"Betina","limit":1,"syntax":["full_text","full_text_simple_match"],"index":"omega:minwiktionary_content","hitsOffset":0,"suggestionRequested":false,"error_message":"index_not_found_exception: no such index"}

So.. We re-run the stuff addWiki.php...

reedy@deploy1001:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki=minwiktionary --baseName=minwiktionary --cluster=eqiad
indexing namespaces...
	Indexing namespaces...done
content index...
	Fetching Elasticsearch version...6.5.4...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-nori, analysis-smartcn, analysis-stconvert
		analysis-stempel, analysis-ukrainian, experimental-highlighter, extra, extra-analysis-esperanto
		extra-analysis-serbian, extra-analysis-slovak, ltr
	Picking analyzer...default
	Inferring index identifier...minwiktionary_content_first
	Creating index...ok
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...different...corrected
	Validating aliases...
		Validating minwiktionary_content alias...alias is free...corrected
		Validating minwiktionary alias...alias not already assigned to this index...corrected
		Updating tracking indexes...done
general index...
	Fetching Elasticsearch version...6.5.4...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-nori, analysis-smartcn, analysis-stconvert
		analysis-stempel, analysis-ukrainian, experimental-highlighter, extra, extra-analysis-esperanto
		extra-analysis-serbian, extra-analysis-slovak, ltr
	Picking analyzer...default
	Inferring index identifier...minwiktionary_general_first
	Creating index...ok
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...different...corrected
	Validating aliases...
		Validating minwiktionary_general alias...alias is free...corrected
		Validating minwiktionary alias...alias not already assigned to this index...corrected
		Updating tracking indexes...done
archive index...
	Fetching Elasticsearch version...6.5.4...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-nori, analysis-smartcn, analysis-stconvert
		analysis-stempel, analysis-ukrainian, experimental-highlighter, extra, extra-analysis-esperanto
		extra-analysis-serbian, extra-analysis-slovak, ltr
	Picking analyzer...default
	Inferring index identifier...minwiktionary_archive_first
	Creating index...ok
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...different...corrected
	Validating aliases...
		Validating minwiktionary_archive alias...alias is free...corrected
		Updating tracking indexes...done
reedy@deploy1001:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki=minwiktionary --baseName=minwiktionary --cluster=codfw
indexing namespaces...
	Indexing namespaces...done
content index...
	Fetching Elasticsearch version...6.5.4...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-nori, analysis-smartcn, analysis-stconvert
		analysis-stempel, analysis-ukrainian, experimental-highlighter, extra, extra-analysis-esperanto
		extra-analysis-serbian, extra-analysis-slovak, ltr
	Picking analyzer...default
	Inferring index identifier...minwiktionary_content_first
	Creating index...ok
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...different...corrected
	Validating aliases...
		Validating minwiktionary_content alias...alias is free...corrected
		Validating minwiktionary alias...alias not already assigned to this index...corrected
		Updating tracking indexes...done
general index...
	Fetching Elasticsearch version...6.5.4...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-nori, analysis-smartcn, analysis-stconvert
		analysis-stempel, analysis-ukrainian, experimental-highlighter, extra, extra-analysis-esperanto
		extra-analysis-serbian, extra-analysis-slovak, ltr
	Picking analyzer...default
	Inferring index identifier...minwiktionary_general_first
	Creating index...ok
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...different...corrected
	Validating aliases...
		Validating minwiktionary_general alias...alias is free...corrected
		Validating minwiktionary alias...alias not already assigned to this index...corrected
		Updating tracking indexes...done
archive index...
	Fetching Elasticsearch version...6.5.4...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-nori, analysis-smartcn, analysis-stconvert
		analysis-stempel, analysis-ukrainian, experimental-highlighter, extra, extra-analysis-esperanto
		extra-analysis-serbian, extra-analysis-slovak, ltr
	Picking analyzer...default
	Inferring index identifier...minwiktionary_archive_first
	Creating index...ok
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...different...corrected
	Validating aliases...
		Validating minwiktionary_archive alias...alias is free...corrected
		Updating tracking indexes...done
reedy@deploy1001:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki=minwiktionary --baseName=minwiktionary --cluster=cloudelastic
indexing namespaces...
	Indexing namespaces...done
content index...
	Fetching Elasticsearch version...6.5.4...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-nori, analysis-smartcn, analysis-stconvert
		analysis-stempel, analysis-ukrainian, experimental-highlighter, extra, extra-analysis-esperanto
		extra-analysis-serbian, extra-analysis-slovak, ltr
	Picking analyzer...default
	Inferring index identifier...minwiktionary_content_first
	Creating index...ok
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...different...corrected
	Validating aliases...
		Validating minwiktionary_content alias...alias is free...corrected
		Validating minwiktionary alias...alias not already assigned to this index...corrected
		Updating tracking indexes...done
general index...
	Fetching Elasticsearch version...6.5.4...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-nori, analysis-smartcn, analysis-stconvert
		analysis-stempel, analysis-ukrainian, experimental-highlighter, extra, extra-analysis-esperanto
		extra-analysis-serbian, extra-analysis-slovak, ltr
	Picking analyzer...default
	Inferring index identifier...minwiktionary_general_first
	Creating index...ok
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...different...corrected
	Validating aliases...
		Validating minwiktionary_general alias...alias is free...corrected
		Validating minwiktionary alias...alias not already assigned to this index...corrected
		Updating tracking indexes...done

It's not erroring now, but no results actually being shown (so not closing). Might need a reindex forcing?

Something is definitely not right. Searching for Adjektiva ("adjective") only gives two results, but visiting random pages will reveal more than that very quickly. Same for Verba ("verb") and Nomina ("noun"). Searching for a known page, like "anak", on the special search page gives contradictory results!

There is a page named "anak" on this wiki.
There were no results matching the query.

Though I believe they have different sources. Clearly the Elastic indexes aren't including everything. This is weird enough that I'm not comfortable just re-indexing it in production without knowing what's going on. But maybe @dcausse or @EBernhardson can take a look.

Of course, we're in the last week of the year (most people will be out most of next week and until Jan 2), so if this isn't a trivial problem to fix, it may not get looked at until we triage tasks in January.

Also, if we can identify the problem (cause or obvious symptom) we can see if other wikis—particularly new ones—have the same problem, and maybe figure out how to prevent it. Sorry not to be more direct help.

The wiki has been reindexed and this should now be resolved. As for the root issue of why the indices were not created in the first place, I'm not sure. Best would be to look at whatever outputs and logs were generated on the first run of addWiki.php, but I'm not sure where to find those. We typically run these kind of maintenance scripts manually as opposed to through a CI/CD system so there is not really a canonical place to find the appropriate run logs.

In T240778#5748261, @EBernhardson wrote:

The wiki has been reindexed and this should now be resolved. As for the root issue of why the indices were not created in the first place, I'm not sure. Best would be to look at whatever outputs and logs were generated on the first run of addWiki.php, but I'm not sure where to find those. We typically run these kind of maintenance scripts manually as opposed to through a CI/CD system so there is not really a canonical place to find the appropriate run logs.

I imagine we still won't have these, at best they might be in a shell backlog, but it's nearly a month ago since

dcausse moved this task from needs triage to Current work on the Discovery-Search board.Jan 2 2020, 2:30 PM

dcausse edited projects, added Discovery-Search (Current work); removed Discovery-Search.

EBernhardson moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Jan 6 2020, 7:28 PM

Gehel closed this task as Resolved.Jan 15 2020, 4:24 PM

Gehel claimed this task.

Aklapper mentioned this in T196740: Rename Wikimedia project tags in Phab (#Wikisource; #Wiktionary) to clarify that they are not about language specific instances?.Mar 30 2020, 1:43 AM

EBernhardson mentioned this in T254331: Suspicious mismatch between psi and omega elastic cluster.Sep 10 2020, 5:04 PM

Dzahn mentioned this in T295192: missing search index on checkuserwiki.Nov 5 2021, 7:40 PM