When running updateSearchIndexConfig.php for test.wikidata, the script chokes on the analyzers
Closed, ResolvedPublic

Description

mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki
content index...
	Fetching Elasticsearch version...1.7.1...ok
	Scanning available plugins...
		langdetect-1.7.0.0-b020e63, wikimedia-extra, analysis-icu, swift-repository, experimental highlighter
		head, whatson
	Infering index identifier...testwikidatawiki_content_1415316749
	Picking analyzer...english
	Index exists so validating...
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...cannot correct
This script encountered an index difference that requires that the index be
copied, indexed to, and then the old index removed. Re-run this script with the
--reindexAndRemoveOk --indexIdentifier=now parameters to do this.

on beta wikidata, I had no problems with this script and locally, normally it's ok.

https://gerrit.wikimedia.org/r/#/c/247788/ is a patch from the Cirrus team that adds an option of updating/validating only the mapping, so could be a work around.

running --reindexAndRemoveOk would probably also be ok in the case of test.wikidata, but if we have the same problem with wikidata then reindex can be more complex and not a great option.

aude created this task.Oct 21 2015, 3:59 PM
aude updated the task description. (Show Details)
aude raised the priority of this task from to High.
aude added projects: Wikidata, CirrusSearch.
aude added a subscriber: aude.
Restricted Application added a project: Discovery. · View Herald TranscriptOct 21 2015, 3:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
aude renamed this task from When running updateSearchIndexCzyonfig.php for test.wikidata, the script chokes on the analyzers to When running updateSearchIndexConfig.php for test.wikidata, the script chokes on the analyzers.Oct 21 2015, 4:00 PM
aude set Security to None.

From the CirrusSearch README:

B.  Build a copy of the index, reindex to it, and then force a full reindex from source documents.  Uses
more disk space but search should be up the entire time:
 php updateSearchIndexConfig.php --reindexAndRemoveOk --indexIdentifier now
 php forceSearchIndex.php

So this seems to be as expected and should work just fine, unless that readme is wrong.

aude added a comment.Oct 23 2015, 3:38 PM

the patch that adds the --justMapping option got split up into two patches, one that adds the option to the script, and the second patch for "allowing mapping customization with numeric fields"

this is the patch that adds the option:

https://gerrit.wikimedia.org/r/#/c/247861/

aude added a comment.Oct 26 2015, 4:16 PM

backported the --justMapping patch and the problem is resolved now

aude closed this task as Resolved.Oct 26 2015, 4:16 PM
aude moved this task from Proposed to Done on the Wikidata-Sprint-2015-10-13 board.