[Story] Use ElasticSearch for entity search on wikidata.org
Closed, ResolvedPublic
Actions

Description

We want to use Elastic for entity search on wikidata.org.

Note: Once that is done there is no need anymore to have aliases, term weights and search keys in the terms table. We could drop those from the production service but we should think about keeping it as a fallback.

Related Objects
Search...

Status	Assigned	Task
Resolved	• Wikidata-bugs	T77898 query taking 10s: TermSqlIndex::getMatchingIDs
Open	None	T46529 Wikidata search problems (tracking)
Resolved	Smalyshev	T78157 [Story] Use ElasticSearch for entity search on wikidata.org
Resolved	Lydia_Pintscher	T88535 [Task] Investigate preconditions & constraints for using Elastic directly
Duplicate	Smalyshev	T117520 Index Wikidata labels, aliases and descriptions as separate fields in ElasticSearch
Resolved	Smalyshev	T125500 [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch
Invalid	None	T132444 [Task] Create repository for WikibaseElastic
Resolved	Smalyshev	T150891 Find a good way to represent multi-lingual text fields in Elastic
Resolved	Smalyshev	T155139 'ContentHandlerForModelID' hook allows creating of handlers that aren't registered
Resolved	Smalyshev	T157604 Make Wikibase register all its content handlers
Resolved	Smalyshev	T157626 ContentHandlerSanityTest assumes every handler can create empty content, but Wikibase can't
Resolved	Smalyshev	T171548 Wikidata Elastic search ignores limit
Resolved	debt	T172422 Add stats marker to wikidata entity search
Resolved	Smalyshev	T172467 Make good prefix search profile for Wikidata entities
Resolved	Smalyshev	T173231 Wikidata Elastic search drops results with matches in different language label
Resolved	Smalyshev	T175741 Set ElasticSearch implementation as default for wbsearchentites on Wikidata
Resolved	Smalyshev	T178851 Use label & description index for fulltext search
Resolved	Smalyshev	T176903 Index wikidata descriptions
Resolved	Smalyshev	T180169 Make list of languages where using stemmed analyzer for Wikibase is beneficial
Resolved	Smalyshev	T182271 Wikidata fulltext search should handle search syntax properly
Resolved	• dcausse	T182293 Tune wikidata fulltext search similarity parameters
Resolved	Smalyshev	T181426 Reindex wikidata to enable description index
Duplicate	None	T117522 Create standard "completion suggestion" API for Search
Declined	None	T120089 Add an internal completion or suggestions API to core SearchEngine
Declined	None	T170392 Create gadget that enables the use of the elastic search backend for the entity selector
Resolved	daniel	T170400 Define metrics for search result quality for the entity selector widget on wikidata.
Resolved	Lydia_Pintscher	T170405 Manually evaluate cirrus based entity search on test system
Open	None	T170547 Metrics to evaluate new search for item suggestor
Declined	None	T170549 Provide A/B test for item suggestor
Resolved	Smalyshev	T162292 Reindex wikidata to pick up labels/descriptions mappings
Resolved	• dcausse	T160926 Make noop script be able replace whole fields with nested subfields
Resolved	• dcausse	T166589 Update wikidata code to take advantage of nested fields noop script
Resolved	Smalyshev	T175199 Index certain statements for Wikidata items

Event Timeline

daniel created this task.Dec 10 2014, 4:32 PM

daniel assigned this task to • Wikidata-bugs.

daniel raised the priority of this task from to High.

daniel updated the task description. (Show Details)

daniel added projects: Performance Issue, Wikidata-Sprint-2014-12-09§, Wikidata.

daniel changed Security from none to None.

daniel added a project: CirrusSearch.

daniel added subscribers: Aklapper, JanZerebecki, daniel, • demon.

Getting the field info into Elastic would be a lot easier if T78011 got implemented.

Lydia_Pintscher added a project: MediaWiki-extensions-WikibaseRepository.Dec 11 2014, 10:36 AM

Lydia_Pintscher moved this task from incoming to needs discussion or investigation on the Wikidata board.Dec 27 2014, 8:07 PM

Lydia_Pintscher removed • Wikidata-bugs as the assignee of this task.Jan 8 2015, 10:39 AM

JanZerebecki added a subtask: T88535: [Task] Investigate preconditions & constraints for using Elastic directly.May 18 2015, 2:11 PM

• Jonas renamed this task from Implement label prefix search based on Elastic (resp Cirrus, Lucene) to [Story] Implement label prefix search based on Elastic (resp Cirrus, Lucene).Aug 13 2015, 4:42 PM

Restricted Application added a project: Discovery-ARCHIVED. · View Herald TranscriptAug 13 2015, 4:42 PM

• demon unsubscribed.Aug 19 2015, 3:43 PM

JanZerebecki added a project: Story.Sep 25 2015, 8:28 PM

Smalyshev added a subtask: T117520: Index Wikidata labels, aliases and descriptions as separate fields in ElasticSearch.Nov 3 2015, 2:00 AM

Smalyshev added a subtask: T117522: Create standard "completion suggestion" API for Search.Nov 3 2015, 2:19 AM

Smalyshev added a subtask: T112028: Implement completion suggester as a Beta Feature.Nov 3 2015, 4:23 AM

• dcausse edited subtasks, added: T120089: Add an internal completion or suggestions API to core SearchEngine; removed: T112028: Implement completion suggester as a Beta Feature.Dec 2 2015, 2:33 PM

• Deskana mentioned this in T110648: [Bug] high-ranking items seemed to have dropped significantly in Special:Search results for wikidata.Feb 3 2016, 6:09 PM

Lydia_Pintscher added a project: Wikidata-Sprint-2016-03-01.Feb 18 2016, 9:51 AM

• dcausse closed subtask T120089: Add an internal completion or suggestions API to core SearchEngine as Declined.Feb 18 2016, 4:38 PM

Tobi_WMDE_SW renamed this task from [Story] Implement label prefix search based on Elastic (resp Cirrus, Lucene) to [Story] Implement entity search API module based on Elastic.Feb 23 2016, 2:37 PM

Tobi_WMDE_SW updated the task description. (Show Details)

Tobi_WMDE_SW updated the task description. (Show Details)Feb 23 2016, 2:41 PM

Tobi_WMDE_SW added a project: Wikidata-Sprint-2016-04-12.Apr 11 2016, 1:52 PM

Restricted Application added a project: Discovery-Search. · View Herald TranscriptApr 11 2016, 1:52 PM

This has not been touched in quite a while; lowering the priority to reflect that reality.

Lydia_Pintscher raised the priority of this task from Low to Medium.Nov 7 2016, 4:10 PM

Liuxinyu970226 mentioned this in T106687: Flow does not support varying language of parts of content based on user interface language (e.g. {{int:}}).Jan 22 2017, 3:20 PM

Lydia_Pintscher closed subtask T88535: [Task] Investigate preconditions & constraints for using Elastic directly as Resolved.Apr 23 2017, 4:33 PM

Liuxinyu970226 subscribed.Apr 25 2017, 2:06 PM

daniel renamed this task from [Story] Implement entity search API module based on Elastic to [Story] Use ElasticSearch for entity search on wikidata.org.Jul 12 2017, 10:43 AM

daniel updated the task description. (Show Details)

daniel created subtask T170392: Create gadget that enables the use of the elastic search backend for the entity selector.Jul 12 2017, 10:50 AM

daniel created subtask T170400: Define metrics for search result quality for the entity selector widget on wikidata..Jul 12 2017, 10:59 AM

daniel created subtask T170405: Manually evaluate cirrus based entity search on test system.Jul 12 2017, 11:11 AM

daniel added a subtask: T170547: Metrics to evaluate new search for item suggestor.Jul 17 2017, 10:37 AM

daniel closed subtask T170400: Define metrics for search result quality for the entity selector widget on wikidata. as Resolved.

debt moved this task from needs triage to This Quarter on the Discovery-Search board.Jul 27 2017, 5:16 PM

Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptJul 27 2017, 5:16 PM

matej_suchanek added a parent task: T46529: Wikidata search problems (tracking).Jul 30 2017, 8:58 AM

Smalyshev added a subtask: T175199: Index certain statements for Wikidata items.Sep 6 2017, 6:41 PM

Lydia_Pintscher closed subtask T170392: Create gadget that enables the use of the elastic search backend for the entity selector as Declined.Sep 13 2017, 6:58 AM

K4-713 subscribed.Sep 15 2017, 11:45 PM