Page MenuHomePhabricator

addWiki failed to create ES indexes
Closed, ResolvedPublic

Description

Trying to do T196747 and I get the below :(

reedy@deploy1001:/srv/mediawiki-staging$ mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki id wikimedia id_internalwikimedia id-internal.wikimedia.org
Creating database id_internalwikimedia for id.wikimedia (Bahasa Indonesia)
Initialising tables
Initialising external storage cluster24...
Initialising external storage cluster25...
Writing main page to Halaman_Utama
Writing sidebar donate link to MediaWiki:Sitesupport-url
indexing namespaces...
	Indexing namespaces...done
content index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_content_first
	Picking analyzer...default
	Creating index...⧼failed to process cluster event (create-index [id_internalwikimedia_content_first], cause [api]) within 30s⧽

Event Timeline

Reedy created this task.Aug 2 2018, 10:27 AM
Restricted Application added subscribers: Petar.petkovic, Base, Aklapper. · View Herald TranscriptAug 2 2018, 10:27 AM
Reedy triaged this task as High priority.Aug 2 2018, 10:28 AM
Reedy updated the task description. (Show Details)
Reedy added a comment.Aug 2 2018, 10:57 AM
reedy@deploy1001:/srv/mediawiki-staging$ mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki id wikimedia id_internalwikimedia id-internal.wikimedia.org
indexing namespaces...
	Indexing namespaces...done
content index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_content_first
	Picking analyzer...default
	Creating index...ok
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...different...

corrected
	Validating aliases...
		Validating id_internalwikimedia_content alias...alias is free...
Unexpected Elasticsearch failure.
Elasticsearch failed in an unexpected way.  This is always a bug in CirrusSearch.
Error type: Elastica\Exception\ResponseException
Message: process_cluster_event_timeout_exception: failed to process cluster event (index-aliases) within 30s
Trace:
#0 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Request.php(193): Elastica\Transport\Http->exec()
#1 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Client.php(674): Elastica\Request->send()
#2 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Client.php(707): Elastica\Client->request()
#3 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Index.php(430): Elastica\Client->requestEndpoint()
#4 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/includes/Maintenance/Validators/SpecificAliasValidator.php(127): Elastica\Index->addAlias()
#5 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/includes/Maintenance/Validators/SpecificAliasValidator.php(79): CirrusSearch\Maintenance\Validators\SpecificAliasValidator->updateFreeIndices()
#6 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/includes/Maintenance/Validators/IndexAliasValidator.php(98): CirrusSearch\Maintenance\Validators\SpecificAliasValidator->updateIndices()
#7 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(419): CirrusSearch\Maintenance\Validators\IndexAliasValidator->validate()
#8 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(384): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateSpecificAlias()
#9 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(245): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateAlias()
#10 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php(64): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->execute()
#11 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/WikimediaMaintenance/addWiki.php(237): CirrusSearch\Maintenance\UpdateSearchIndexConfig->execute()
#12 /srv/mediawiki-staging/php-1.32.0-wmf.14/maintenance/doMaintenance.php(94): AddWiki->execute()
#13 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/WikimediaMaintenance/addWiki.php(371): include()
#14 /srv/mediawiki-staging/multiversion/MWScript.php(100): include()
#15 {main}
Reedy added a comment.Aug 2 2018, 11:12 AM

It seems $wgCirrusSearchMasterTimeout isn't being respected, as things are timing out at 30s

And another...

reedy@deploy1001:/srv/mediawiki-staging$ mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki id wikimedia id_internalwikimedia id-internal.wikimedia.org
indexing namespaces...
	Indexing namespaces...done
content index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_content_first
	Picking analyzer...default
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...ok
	Validating aliases...
		Validating id_internalwikimedia_content alias...ok
		Validating id_internalwikimedia alias...ok
		Updating tracking indexes...done
general index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_general_first
	Picking analyzer...default
	Creating index...ok
		Validating number of shards...
Unexpected Elasticsearch failure.
Elasticsearch failed in an unexpected way.  This is always a bug in CirrusSearch.
Error type: Elastica\Exception\ResponseException
Message: index_not_found_exception: no such index
Trace:
#0 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Request.php(193): Elastica\Transport\Http->exec()
#1 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Client.php(674): Elastica\Request->send()
#2 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Index.php(547): Elastica\Client->request()
#3 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Index/Settings.php(383): Elastica\Index->request()
#4 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Index/Settings.php(74): Elastica\Index\Settings->request()
#5 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/includes/Maintenance/Validators/NumberOfShardsValidator.php(38): Elastica\Index\Settings->get()
#6 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(344): CirrusSearch\Maintenance\Validators\NumberOfShardsValidator->validate()
#7 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(294): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateIndexSettings()
#8 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(242): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateIndex()
#9 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php(64): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->execute()
#10 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/WikimediaMaintenance/addWiki.php(238): CirrusSearch\Maintenance\UpdateSearchIndexConfig->execute()
#11 /srv/mediawiki-staging/php-1.32.0-wmf.14/maintenance/doMaintenance.php(94): AddWiki->execute()
#12 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/WikimediaMaintenance/addWiki.php(372): include()
#13 /srv/mediawiki-staging/multiversion/MWScript.php(100): include()
#14 {main}
Reedy added a comment.Aug 2 2018, 11:16 AM
reedy@deploy1001:/srv/mediawiki-staging$ mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki id wikimedia id_internalwikimedia id-internal.wikimedia.org
indexing namespaces...
	Indexing namespaces...done
content index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_content_first
	Picking analyzer...default
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...ok
	Validating aliases...
		Validating id_internalwikimedia_content alias...ok
		Validating id_internalwikimedia alias...ok
		Updating tracking indexes...done
general index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_general_first
	Picking analyzer...default
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...different...^[[Acorrected
	Validating aliases...
		Validating id_internalwikimedia_general alias...alias is free...corrected
		Validating id_internalwikimedia alias...alias not already assigned to this index...
Unexpected Elasticsearch failure.
Elasticsearch failed in an unexpected way.  This is always a bug in CirrusSearch.
Error type: Elastica\Exception\ResponseException
Message: process_cluster_event_timeout_exception: failed to process cluster event (index-aliases) within 30s
Trace:
#0 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Request.php(193): Elastica\Transport\Http->exec()
#1 /srv/mediawiki-staging/php-1.32.0-wmf.14/vendor/ruflin/elastica/lib/Elastica/Client.php(674): Elastica\Request->send()
#2 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/includes/Maintenance/Validators/IndexAllAliasValidator.php(49): Elastica\Client->request()
#3 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/includes/Maintenance/Validators/IndexAliasValidator.php(98): CirrusSearch\Maintenance\Validators\IndexAllAliasValidator->updateIndices()
#4 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(428): CirrusSearch\Maintenance\Validators\IndexAliasValidator->validate()
#5 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(385): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateAllAlias()
#6 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateOneSearchIndexConfig.php(245): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->validateAlias()
#7 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php(64): CirrusSearch\Maintenance\UpdateOneSearchIndexConfig->execute()
#8 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/WikimediaMaintenance/addWiki.php(238): CirrusSearch\Maintenance\UpdateSearchIndexConfig->execute()
#9 /srv/mediawiki-staging/php-1.32.0-wmf.14/maintenance/doMaintenance.php(94): AddWiki->execute()
#10 /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/WikimediaMaintenance/addWiki.php(372): include()
#11 /srv/mediawiki-staging/multiversion/MWScript.php(100): include()
#12 {main}
Reedy added a comment.Aug 2 2018, 11:17 AM

And eventually...

reedy@deploy1001:/srv/mediawiki-staging$ mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki id wikimedia id_internalwikimedia id-internal.wikimedia.org
indexing namespaces...
	Indexing namespaces...done
content index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_content_first
	Picking analyzer...default
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...ok
	Validating aliases...
		Validating id_internalwikimedia_content alias...ok
		Validating id_internalwikimedia alias...ok
		Updating tracking indexes...done
general index...
	Fetching Elasticsearch version...5.5.2...ok
	Scanning available plugins...
		analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
		analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
		ltr
	Inferring index identifier...id_internalwikimedia_general_first
	Picking analyzer...default
		Validating number of shards...ok
		Validating replica range...ok
		Validating shard allocation settings...done
		Validating max shards per node...ok
	Validating analyzers...ok
	Validating mappings...
		Validating mapping...ok
	Validating aliases...
		Validating id_internalwikimedia_general alias...ok
		Validating id_internalwikimedia alias...alias not already assigned to this index...corrected
		Updating tracking indexes...done
	indexing namespaces...
		Indexing namespaces...done
	content index...
		Fetching Elasticsearch version...5.5.2...ok
		Scanning available plugins...
			analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
			analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
			ltr
		Inferring index identifier...id_internalwikimedia_content_first
		Picking analyzer...default
		Creating index...ok
			Validating number of shards...ok
			Validating replica range...ok
			Validating shard allocation settings...done
			Validating max shards per node...ok
		Validating analyzers...ok
		Validating mappings...
			Validating mapping...different...corrected
		Validating aliases...
			Validating id_internalwikimedia_content alias...alias is free...corrected
			Validating id_internalwikimedia alias...alias not already assigned to this index...corrected
			Updating tracking indexes...done
	general index...
		Fetching Elasticsearch version...5.5.2...ok
		Scanning available plugins...
			analysis-hebrew, analysis-icu, analysis-smartcn, analysis-stconvert, analysis-stempel
			analysis-ukrainian, experimental-highlighter, extra, extra-analysis-serbian, extra-analysis-slovak
			ltr
		Inferring index identifier...id_internalwikimedia_general_first
		Picking analyzer...default
		Creating index...ok
			Validating number of shards...ok
			Validating replica range...ok
			Validating shard allocation settings...done
			Validating max shards per node...ok
		Validating analyzers...ok
		Validating mappings...
			Validating mapping...different...corrected
		Validating aliases...
			Validating id_internalwikimedia_general alias...alias is free...corrected
			Validating id_internalwikimedia alias...alias not already assigned to this index...corrected
			Updating tracking indexes...done
done.
Making sure mwstore://local-multiwrite/local-public exists...making 'mwstore://local-multiwrite/local-public' public...done.
Making sure mwstore://local-multiwrite/local-thumb exists...making 'mwstore://local-multiwrite/local-thumb' public...done.
Making sure mwstore://local-multiwrite/local-transcoded exists...making 'mwstore://local-multiwrite/local-transcoded' public...done.
Making sure mwstore://local-multiwrite/local-temp exists...making 'mwstore://local-multiwrite/local-temp' private...done.
Making sure mwstore://local-multiwrite/local-deleted exists...making 'mwstore://local-multiwrite/local-deleted' private...done.
Making sure mwstore://local-multiwrite/timeline-render exists...making 'mwstore://local-multiwrite/timeline-render' public...done.
[Thu Aug  2 11:16:42 2018] [hphp] [14014:7fad457203c0:0:000001] [] 
Notice: Undefined variable: ucSiteGroup in /srv/mediawiki-staging/php-1.32.0-wmf.14/extensions/WikimediaMaintenance/addWiki.php on line 284
Done. sync the config as in https://wikitech.wikimedia.org/wiki/Add_a_wiki#MediaWiki_configuration
reedy@deploy1001:/srv/mediawiki-staging$
EBernhardson closed this task as Resolved.
EBernhardson claimed this task.
EBernhardson added a subscriber: EBernhardson.

I believe the underlying issue, timeouts from the elasticsearch master while trying to mutate the cluster state, have been resolved.