Page MenuHomePhabricator
Create Task
Maniphest T200994

addWiki failed to create ES indexes
Closed, ResolvedPublic
Actions

Description

Trying to do T196747 and I get the below :(

reedy@deploy1001:/srv/mediawiki-staging$ mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki id wikimedia id_internalwikimedia id-internal.wikimedia.org
Creating database id_internalwikimedia for id.wikimedia (Bahasa Indonesia)
Initialising tables
Initialising external storage cluster24...
Initialising external storage cluster25...
Writing main page to Halaman_Utama
Writing sidebar donate link to MediaWiki:Sitesupport-url
indexing namespaces...
	Indexing namespaces...done
content index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_content_first
	Picking analyzer...default
	Creating index...⧼failed to process cluster event (create-index [id_internalwikimedia_content_first], cause [api]) within 30s⧽

Related Objects
Search...

Event Timeline

Reedy created this task.Aug 2 2018, 10:27 AM
Restricted Application added subscribers: • Petar.petkovic, Base, Aklapper. · View Herald TranscriptAug 2 2018, 10:27 AM
Reedy triaged this task as High priority.Aug 2 2018, 10:28 AM
Reedy updated the task description. (Show Details)
Reedy added a comment.Aug 2 2018, 10:57 AM
reedy@deploy1001:/srv/mediawiki-staging$ mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki id wikimedia id_internalwikimedia id-internal.wikimedia.org
indexing namespaces...
	Indexing namespaces...done
content index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_content_first
	Picking analyzer...default
	Creating index...ok
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...different...

corrected
	Validating aliases...
		Validating id_internalwikimedia_content alias...alias is free...
Unexpected Elasticsearch failure.
Elasticsearch failed in an unexpected way.  This is always a bug in CirrusSearch.
Error type: Elastica\Exception\ResponseException
Message: process_cluster_event_timeout_exception: failed to process cluster event (index-aliases) within 30s
Trace:
#0 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Request.php(193): Elastica\Transport\Http->exec()
#1 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Client.php(674): Elastica\Request->send()
#2 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Client.php(707): Elastica\Client->request()
#3 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Index.php(430): Elastica\Client->requestEndpoint()
#4 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/includes/Maintenance/Validators/SpecificAliasValidator.php(127): Elastica\Index->addAlias()
#5 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/includes/Maintenance/Validators/SpecificAliasValidator.php(79): CirrusSearch\Maintenance\Validators\SpecificAliasValidator->updateFreeIndices()
#6 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/includes/Maintenance/Validators/IndexAliasValidator.php(98): CirrusSearch\Maintenance\Validators\SpecificAliasValidator->updateIndices()
#7 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(419): CirrusSearch\Maintenance\Validators\IndexAliasValidator->validate()
#8 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(384): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateSpecificAlias()
#9 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(245): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateAlias()
#10 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php(64): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->execute()
#11 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/WikimediaMaintenance/addWiki.php(237): CirrusSearch\Maintenance\UpdateSearchIndexConfig->execute()
#12 /srv/mediawiki-staging/php-1.32.0-wmf.14/maintenance/doMaintenance.php(94): AddWiki->execute()
#13 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/WikimediaMaintenance/addWiki.php(371): include()
#14 /srv/mediawiki-staging/multiversion/MWScript.php(100): include()
#15 {main}
Jayprakash12345 subscribed.Aug 2 2018, 10:58 AM
Urbanecm mentioned this in T199509: Create Wikimania wiki.Aug 2 2018, 11:05 AM
Urbanecm subscribed.
Reedy added a comment.Aug 2 2018, 11:12 AM

It seems $wgCirrusSearchMasterTimeout isn't being respected, as things are timing out at 30s

And another...

reedy@deploy1001:/srv/mediawiki-staging$ mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki id wikimedia id_internalwikimedia id-internal.wikimedia.org
indexing namespaces...
	Indexing namespaces...done
content index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_content_first
	Picking analyzer...default
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...ok
	Validating aliases...
		Validating id_internalwikimedia_content alias...ok
		Validating id_internalwikimedia alias...ok
		Updating tracking indexes...done
general index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_general_first
	Picking analyzer...default
	Creating index...ok
		Validating number of shards...
Unexpected Elasticsearch failure.
Elasticsearch failed in an unexpected way.  This is always a bug in CirrusSearch.
Error type: Elastica\Exception\ResponseException
Message: index_not_found_exception: no such index
Trace:
#0 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Request.php(193): Elastica\Transport\Http->exec()
#1 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Client.php(674): Elastica\Request->send()
#2 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Index.php(547): Elastica\Client->request()
#3 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Index/Settings.php(383): Elastica\Index->request()
#4 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Index/Settings.php(74): Elastica\Index\Settings->request()
#5 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/includes/Maintenance/Validators/NumberOfShardsValidator.php(38): Elastica\Index\Settings->get()
#6 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(344): CirrusSearch\Maintenance\Validators\NumberOfShardsValidator->validate()
#7 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(294): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateIndexSettings()
#8 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(242): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateIndex()
#9 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php(64): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->execute()
#10 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/WikimediaMaintenance/addWiki.php(238): CirrusSearch\Maintenance\UpdateSearchIndexConfig->execute()
#11 /srv/mediawiki-staging/php-1.32.0-wmf.14/maintenance/doMaintenance.php(94): AddWiki->execute()
#12 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/WikimediaMaintenance/addWiki.php(372): include()
#13 /srv/mediawiki-staging/multiversion/MWScript.php(100): include()
#14 {main}
Reedy added a comment.Aug 2 2018, 11:16 AM
reedy@deploy1001:/srv/mediawiki-staging$ mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki id wikimedia id_internalwikimedia id-internal.wikimedia.org
indexing namespaces...
	Indexing namespaces...done
content index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_content_first
	Picking analyzer...default
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...ok
	Validating aliases...
		Validating id_internalwikimedia_content alias...ok
		Validating id_internalwikimedia alias...ok
		Updating tracking indexes...done
general index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_general_first
	Picking analyzer...default
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...different...^[[Acorrected
	Validating aliases...
		Validating id_internalwikimedia_general alias...alias is free...corrected
		Validating id_internalwikimedia alias...alias not already assigned to this index...
Unexpected Elasticsearch failure.
Elasticsearch failed in an unexpected way.  This is always a bug in CirrusSearch.
Error type: Elastica\Exception\ResponseException
Message: process_cluster_event_timeout_exception: failed to process cluster event (index-aliases) within 30s
Trace:
#0 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Request.php(193): Elastica\Transport\Http->exec()
#1 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Client.php(674): Elastica\Request->send()
#2 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/includes/Maintenance/Validators/IndexAllAliasValidator.php(49): Elastica\Client->request()
#3 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/includes/Maintenance/Validators/IndexAliasValidator.php(98): CirrusSearch\Maintenance\Validators\IndexAllAliasValidator->updateIndices()
#4 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(428): CirrusSearch\Maintenance\Validators\IndexAliasValidator->validate()
#5 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(385): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateAllAlias()
#6 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(245): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateAlias()
#7 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php(64): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->execute()
#8 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/WikimediaMaintenance/addWiki.php(238): CirrusSearch\Maintenance\UpdateSearchIndexConfig->execute()
#9 /srv/mediawiki-staging/php-1.32.0-wmf.14/maintenance/doMaintenance.php(94): AddWiki->execute()
#10 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/WikimediaMaintenance/addWiki.php(372): include()
#11 /srv/mediawiki-staging/multiversion/MWScript.php(100): include()
#12 {main}
Reedy added a comment.Aug 2 2018, 11:17 AM

And eventually...

reedy@deploy1001:/srv/mediawiki-staging$ mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki id wikimedia id_internalwikimedia id-internal.wikimedia.org
indexing namespaces...
	Indexing namespaces...done
content index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_content_first
	Picking analyzer...default
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...ok
	Validating aliases...
		Validating id_internalwikimedia_content alias...ok
		Validating id_internalwikimedia alias...ok
		Updating tracking indexes...done
general index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_general_first
	Picking analyzer...default
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...ok
	Validating aliases...
		Validating id_internalwikimedia_general alias...ok
		Validating id_internalwikimedia alias...alias not already assigned to this index...corrected
		Updating tracking indexes...done
	indexing namespaces...
		Indexing namespaces...done
	content index...
		Fetching Elasticsearch version...5.5.2...ok
		Scanning available plugins...
			analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
			analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
			ltr
		Inferring index identifier...id_internalwikimedia_content_first
		Picking analyzer...default
		Creating index...ok
			Validating number of shards...ok
			Validating replica range...ok
			Validating shard allocation settings...done
			Validating max shards per node...ok
		Validating analyzers...ok
		Validating mappings...
			Validating mapping...different...corrected
		Validating aliases...
			Validating id_internalwikimedia_content alias...alias is free...corrected
			Validating id_internalwikimedia alias...alias not already assigned to this index...corrected
			Updating tracking indexes...done
	general index...
		Fetching Elasticsearch version...5.5.2...ok
		Scanning available plugins...
			analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
			analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
			ltr
		Inferring index identifier...id_internalwikimedia_general_first
		Picking analyzer...default
		Creating index...ok
			Validating number of shards...ok
			Validating replica range...ok
			Validating shard allocation settings...done
			Validating max shards per node...ok
		Validating analyzers...ok
		Validating mappings...
			Validating mapping...different...corrected
		Validating aliases...
			Validating id_internalwikimedia_general alias...alias is free...corrected
			Validating id_internalwikimedia alias...alias not already assigned to this index...corrected
			Updating tracking indexes...done
done.
Making sure mwstore://local-multiwrite/local-public exists...making 'mwstore://local-multiwrite/local-public' public...done.
Making sure mwstore://local-multiwrite/local-thumb exists...making 'mwstore://local-multiwrite/local-thumb' public...done.
Making sure mwstore://local-multiwrite/local-transcoded exists...making 'mwstore://local-multiwrite/local-transcoded' public...done.
Making sure mwstore://local-multiwrite/local-temp exists...making 'mwstore://local-multiwrite/local-temp' private...done.
Making sure mwstore://local-multiwrite/local-deleted exists...making 'mwstore://local-multiwrite/local-deleted' private...done.
Making sure mwstore://local-multiwrite/timeline-render exists...making 'mwstore://local-multiwrite/timeline-render' public...done.
[Thu Aug  2 11:16:42 2018] [hphp] [14014:7fad457203c0:0:000001] [] 
Notice: Undefined variable: ucSiteGroup in /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/WikimediaMaintenance/addWiki.php on line 284
Done. sync the config as in https://wikitech.wikimedia.org/wiki/Add_a_wiki#MediaWiki_configuration
reedy@deploy1001:/srv/mediawiki-staging$
• Petar.petkovic unsubscribed.Aug 2 2018, 12:10 PM
debt edited projects, added Discovery-Search (Current work); removed Discovery-Search.Aug 2 2018, 5:14 PM
EBernhardson moved this task from Incoming to Waiting on the Discovery-Search (Current work) board.Sep 10 2018, 5:42 PM
EBernhardson closed this task as Resolved.Feb 4 2019, 8:59 PM
EBernhardson claimed this task.
EBernhardson closed subtask T200999: wgCirrusSearchMasterTimeout is not passed to all request during updateOneSearchIndexConfig as Declined.
EBernhardson subscribed.

I believe the underlying issue, timeouts from the elasticsearch master while trying to mutate the cluster state, have been resolved.

EBernhardson moved this task from Waiting to Needs Reporting on the Discovery-Search (Current work) board.May 6 2019, 3:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL