[EPIC] Create a dedicated elasticsearch cluster for Commons and Wikidata
Open, MediumPublic
Actions

Assigned To

None

Authored By

	Gehel
	Oct 15 2020, 2:59 PM

Description

As a developer / operator of Search, I want to have better visibility on large indices, be able to isolate their impact to other search indices and tune the resources according to specific needs.

Wikibase related indices are larger and growing faster than our other indices. This presents an issue for resources allocation. Isolating those indices on a specific cluster should allow better visibility, resources management / planning.

Some investigation is needed to decide what the optimal solution is here (another cluster on the same physical hosts, how to split the hardware resources, moving some smaller indices to Ganeti, other solutions...).

AC:

Commons and Wikibase indices are served from a dedicated elasticsearch cluster.

Related Objects
Search...

Status	Assigned	Task
Resolved	Gehel	T247556 [Epic] Search platform - Hardware requests for 2020-2021
Open	None	T265621 [EPIC] Create a dedicated elasticsearch cluster for Commons and Wikidata
Resolved	EBernhardson	T265914 Investigate Resource Needs for Commons and Wikidata Elasticsearch indices
Resolved	EBernhardson	T265931 Set up dashboard to track resource usage for Commons and Wikidata Elasticsearch indexes

Event Timeline

Gehel created this task.Oct 15 2020, 2:59 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 15 2020, 2:59 PM

• Zbyszko renamed this task from Create a dedicated elasticsearch cluster for Commons and Wikidata to [EPIC]Create a dedicated elasticsearch cluster for Commons and Wikidata.Oct 19 2020, 3:25 PM

TJones renamed this task from [EPIC]Create a dedicated elasticsearch cluster for Commons and Wikidata to [EPIC] Create a dedicated elasticsearch cluster for Commons and Wikidata.Oct 19 2020, 3:25 PM

• Zbyszko added a subtask: T265914: Investigate Resource Needs for Commons and Wikidata Elasticsearch indices .Oct 19 2020, 3:28 PM

CBogen moved this task from needs triage to Current work on the Discovery-Search board.Oct 26 2020, 4:25 PM

CBogen edited projects, added Discovery-Search (Current work); removed Discovery-Search.

CBogen merged a task: T264632: [Epic] Longer term plan for increases in Elasticsearch cluster disk IO.Oct 26 2020, 4:31 PM

CBogen added a subscriber: EBernhardson.

CBogen moved this task from Incoming to Epics on the Discovery-Search (Current work) board.Oct 26 2020, 6:37 PM

CBogen mentioned this in T271493: Implement 50kb limit on file text indexing for to reduce increasing commonswiki_file on-disk size.Jan 13 2021, 8:01 PM

CBogen edited projects, added Discovery-Search; removed Discovery-Search (Current work).Jan 21 2021, 3:04 PM

CBogen triaged this task as Medium priority.Jan 25 2021, 4:23 PM

CBogen added a parent task: T247556: [Epic] Search platform - Hardware requests for 2020-2021.

CBogen moved this task from needs triage to elastic / cirrus on the Discovery-Search board.

Gehel closed subtask T265914: Investigate Resource Needs for Commons and Wikidata Elasticsearch indices as Resolved.Mar 24 2021, 1:28 PM