Page MenuHomePhabricator
Create Task
Maniphest T107348

Timeouts when trying to create mappings.
Closed, ResolvedPublic
Actions

Description

There is a default master_timeout of 30s, we are running over that at time preventing index creation. This has happened both when creating the indexes for a new wiki and again today creating indexes for the titlesuggest feature. We can override the timeout by setting master_timeout in the query string, this is not supported by the Elastica client library we use in all instances.

Ideally we could fix this by making the cluster respond in < 30s, but i fear we might need to make the timeout adjustable until then.

Details

SubjectRepoBranchLines +/-
Utilize the master timeout for creation and deletion as wellmediawiki/extensions/CirrusSearchmaster+37 -4
Add $wgCirrusSearchMasterTimeout to deal with master slowdownsmediawiki/extensions/CirrusSearchwmf/1.27.0-wmf.8+45 -4
Add $wgCirrusSearchMasterTimeout to deal with master slowdownsmediawiki/extensions/CirrusSearchwmf/1.27.0-wmf.7+45 -4
Add $wgCirrusSearchMasterTimeout to deal with master slowdownsmediawiki/extensions/CirrusSearchmaster+45 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

EBernhardson created this task.Jul 29 2015, 9:23 PM
EBernhardson raised the priority of this task from to Needs Triage.
EBernhardson updated the task description. (Show Details)
EBernhardson added a project: CirrusSearch.
EBernhardson subscribed.
Restricted Application added a project: Discovery-ARCHIVED. · View Herald TranscriptJul 29 2015, 9:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Ironholds moved this task from Needs triage to Search on the Discovery-ARCHIVED board.Aug 4 2015, 8:16 AM
EBernhardson added a comment.Aug 31 2015, 8:48 PM

This requires adding adding master_timeout=60s to the query string which is not directly supported by the
Mapping class we use. Going to require minor hacks all around for temporary solution, or coming up with a higher level way of attaching master_timeout to the right kinds of requests.

EBernhardson renamed this task from Add an option to set timeouts from cli in updateSearchIndexConfig.php to Timeouts when trying to create mappings..Sep 1 2015, 12:27 AM
EBernhardson updated the task description. (Show Details)
EBernhardson set Security to None.
EBernhardson added a parent task: T109089: EPIC: Cultivating the Elasticsearch garden (operational lessons from 1.7.1 upgrade).
• ksmith added a project: Essential-Work.Oct 8 2015, 11:59 PM
gerritbot subscribed.Dec 9 2015, 11:36 PM

Change 258017 had a related patch set uploaded (by EBernhardson):
Add $wgCirrusSearchMasterTimeout to deal with master slowdowns

https://gerrit.wikimedia.org/r/258017

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptDec 9 2015, 11:36 PM
gerritbot added a project: Patch-For-Review.Dec 9 2015, 11:36 PM
EBernhardson added a comment.Dec 9 2015, 11:37 PM

I've also submitted a fix to the Elastica library which will let us change the master timeout in a less hacky way. This has been merged and will be in the next release: https://github.com/ruflin/Elastica/pull/1004

EBernhardson claimed this task.Dec 9 2015, 11:57 PM
EBernhardson added a project: Discovery-Search (Current work).
EBernhardson moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.
gerritbot added a comment.Dec 10 2015, 12:48 AM

Change 258017 merged by jenkins-bot:
Add $wgCirrusSearchMasterTimeout to deal with master slowdowns

https://gerrit.wikimedia.org/r/258017

ReleaseTaggerBot added a project: MW-1.27-release (WMF-deploy-2015-12-15_(1.27.0-wmf.9)).Dec 10 2015, 1:00 AM
gerritbot added a comment.Dec 10 2015, 1:15 AM

Change 258088 had a related patch set uploaded (by EBernhardson):
Add $wgCirrusSearchMasterTimeout to deal with master slowdowns

https://gerrit.wikimedia.org/r/258088

gerritbot added a comment.Dec 10 2015, 1:16 AM

Change 258089 had a related patch set uploaded (by EBernhardson):
Add $wgCirrusSearchMasterTimeout to deal with master slowdowns

https://gerrit.wikimedia.org/r/258089

gerritbot added a comment.Dec 10 2015, 1:19 AM

Change 258088 merged by jenkins-bot:
Add $wgCirrusSearchMasterTimeout to deal with master slowdowns

https://gerrit.wikimedia.org/r/258088

gerritbot added a comment.Dec 10 2015, 1:21 AM

Change 258089 merged by jenkins-bot:
Add $wgCirrusSearchMasterTimeout to deal with master slowdowns

https://gerrit.wikimedia.org/r/258089

EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Dec 10 2015, 1:25 AM
ReleaseTaggerBot added projects: MW-1.27-release (WMF-deploy-2015-11-17_(1.27.0-wmf.7)), MW-1.27-release (WMF-deploy-2015-12-08_(1.27.0-wmf.8)).Dec 10 2015, 2:00 AM
gerritbot added a comment.Dec 10 2015, 3:09 AM

Change 258095 had a related patch set uploaded (by EBernhardson):
Utilize the master timeout for creation and deletion as well

https://gerrit.wikimedia.org/r/258095

gerritbot added a comment.Dec 10 2015, 6:32 AM

Change 258095 merged by jenkins-bot:
Utilize the master timeout for creation and deletion as well

https://gerrit.wikimedia.org/r/258095

• Deskana triaged this task as Medium priority.Dec 17 2015, 11:46 PM
• Deskana moved this task from Search to On Sprint Board on the Discovery-ARCHIVED board.
• Deskana subscribed.
• Deskana closed this task as Resolved.Dec 17 2015, 11:50 PM
• Deskana moved this task from Inbox to Resolved/Invalid/Declined/Legacy on the CirrusSearch board.Dec 31 2015, 5:08 AM
• Deskana moved this task from Needs Reporting to Resolved on the Discovery-Search (Current work) board.Jan 28 2016, 6:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL