Page MenuHomePhabricator

Timeouts when trying to create mappings.
Closed, ResolvedPublic

Description

There is a default master_timeout of 30s, we are running over that at time preventing index creation. This has happened both when creating the indexes for a new wiki and again today creating indexes for the titlesuggest feature. We can override the timeout by setting master_timeout in the query string, this is not supported by the Elastica client library we use in all instances.

Ideally we could fix this by making the cluster respond in < 30s, but i fear we might need to make the timeout adjustable until then.

Details

Related Gerrit Patches:
mediawiki/extensions/CirrusSearch : masterUtilize the master timeout for creation and deletion as well
mediawiki/extensions/CirrusSearch : wmf/1.27.0-wmf.8Add $wgCirrusSearchMasterTimeout to deal with master slowdowns
mediawiki/extensions/CirrusSearch : wmf/1.27.0-wmf.7Add $wgCirrusSearchMasterTimeout to deal with master slowdowns
mediawiki/extensions/CirrusSearch : masterAdd $wgCirrusSearchMasterTimeout to deal with master slowdowns

Event Timeline

EBernhardson raised the priority of this task from to Needs Triage.
EBernhardson updated the task description. (Show Details)
EBernhardson added a project: CirrusSearch.
EBernhardson added a subscriber: EBernhardson.
Restricted Application added a project: Discovery. · View Herald TranscriptJul 29 2015, 9:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Ironholds moved this task from Needs triage to Search on the Discovery board.Aug 4 2015, 8:16 AM

This requires adding adding master_timeout=60s to the query string which is not directly supported by the
Mapping class we use. Going to require minor hacks all around for temporary solution, or coming up with a higher level way of attaching master_timeout to the right kinds of requests.

EBernhardson renamed this task from Add an option to set timeouts from cli in updateSearchIndexConfig.php to Timeouts when trying to create mappings..Sep 1 2015, 12:27 AM
EBernhardson updated the task description. (Show Details)
EBernhardson set Security to None.

Change 258017 had a related patch set uploaded (by EBernhardson):
Add $wgCirrusSearchMasterTimeout to deal with master slowdowns

https://gerrit.wikimedia.org/r/258017

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptDec 9 2015, 11:36 PM

I've also submitted a fix to the Elastica library which will let us change the master timeout in a less hacky way. This has been merged and will be in the next release: https://github.com/ruflin/Elastica/pull/1004

Change 258017 merged by jenkins-bot:
Add $wgCirrusSearchMasterTimeout to deal with master slowdowns

https://gerrit.wikimedia.org/r/258017

Change 258088 had a related patch set uploaded (by EBernhardson):
Add $wgCirrusSearchMasterTimeout to deal with master slowdowns

https://gerrit.wikimedia.org/r/258088

Change 258089 had a related patch set uploaded (by EBernhardson):
Add $wgCirrusSearchMasterTimeout to deal with master slowdowns

https://gerrit.wikimedia.org/r/258089

Change 258088 merged by jenkins-bot:
Add $wgCirrusSearchMasterTimeout to deal with master slowdowns

https://gerrit.wikimedia.org/r/258088

Change 258089 merged by jenkins-bot:
Add $wgCirrusSearchMasterTimeout to deal with master slowdowns

https://gerrit.wikimedia.org/r/258089

Change 258095 had a related patch set uploaded (by EBernhardson):
Utilize the master timeout for creation and deletion as well

https://gerrit.wikimedia.org/r/258095

Change 258095 merged by jenkins-bot:
Utilize the master timeout for creation and deletion as well

https://gerrit.wikimedia.org/r/258095

Deskana triaged this task as Medium priority.Dec 17 2015, 11:46 PM
Deskana moved this task from Search to On Sprint Board on the Discovery board.
Deskana added a subscriber: Deskana.
Deskana closed this task as Resolved.Dec 17 2015, 11:50 PM