Make good prefix search profile for Wikidata entities
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Smalyshev
	Aug 4 2017, 12:32 AM

Description

We need to create a profile that provides good results for prefix search.

Current profiles have these issues:

Over-boosting label counts
Not counting incoming links
Disambig pages shown way too high (T148411)
Rescore window of 8192 probably way too large

Testing site: http://elastic-wikidata.wmflabs.org/wb.html

Details

	Subject	Repo	Branch	Lines +/-
	Tune Wikidata entity search parameters	mediawiki/extensions/Wikibase	master	+1 K -229

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Wikidata-bugs	T77898 query taking 10s: TermSqlIndex::getMatchingIDs
Open	None	T46529 Wikidata search problems (tracking)
Resolved	aude	T110648 [Bug] high-ranking items seemed to have dropped significantly in Special:Search results for wikidata
Resolved	Smalyshev	T119067 Adjust rescoring config for Wikidata to consider sitelink count
Resolved	aude	T119066 Add sitelink count to search index for Wikidata
Resolved	None	T120217 [Task] Add CirrusSearch and Elastica as dependencies of Wikibase on Jenkins
Invalid	None	T161334 Stemming for item suggestions, e.g. "the" vs. no "the" on Wikidata
Resolved	Smalyshev	T78157 [Story] Use ElasticSearch for entity search on wikidata.org
Resolved	Lydia_Pintscher	T88535 [Task] Investigate preconditions & constraints for using Elastic directly
Duplicate	Smalyshev	T117520 Index Wikidata labels, aliases and descriptions as separate fields in ElasticSearch
Duplicate	None	T117522 Create standard "completion suggestion" API for Search
Declined	None	T120089 Add an internal completion or suggestions API to core SearchEngine
Declined	None	T170392 Create gadget that enables the use of the elastic search backend for the entity selector
Resolved	daniel	T170400 Define metrics for search result quality for the entity selector widget on wikidata.
Resolved	Lydia_Pintscher	T170405 Manually evaluate cirrus based entity search on test system
Open	None	T170547 Metrics to evaluate new search for item suggestor
Declined	None	T170549 Provide A/B test for item suggestor
Resolved	Smalyshev	T162292 Reindex wikidata to pick up labels/descriptions mappings
Resolved	Smalyshev	T175199 Index certain statements for Wikidata items
Declined	None	T141813 Add full-text search support to Query Service
Resolved	Smalyshev	T125500 [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch
Resolved	dcausse	T160926 Make noop script be able replace whole fields with nested subfields
Resolved	dcausse	T166589 Update wikidata code to take advantage of nested fields noop script
Resolved	Smalyshev	T172467 Make good prefix search profile for Wikidata entities
Resolved	Smalyshev	T178851 Use label & description index for fulltext search
Resolved	Smalyshev	T176903 Index wikidata descriptions
Resolved	Smalyshev	T180169 Make list of languages where using stemmed analyzer for Wikibase is beneficial
Resolved	Smalyshev	T182271 Wikidata fulltext search should handle search syntax properly
Resolved	dcausse	T182293 Tune wikidata fulltext search similarity parameters

Event Timeline

Smalyshev created this task.Aug 4 2017, 12:32 AM

Restricted Application added subscribers: PokestarFan, Aklapper. · View Herald TranscriptAug 4 2017, 12:32 AM

Smalyshev claimed this task.Aug 4 2017, 12:33 AM

Smalyshev triaged this task as Medium priority.

Smalyshev added a parent task: T125500: [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch.

Smalyshev moved this task from needs triage to Current work on the Discovery-Search board.

Smalyshev edited projects, added Discovery-Search (Current work); removed Discovery-Search.

Smalyshev moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.

Smalyshev updated the task description. (Show Details)

Smalyshev added a project: Discovery-ARCHIVED.Aug 4 2017, 12:41 AM

Smalyshev added a project: User-Smalyshev.Aug 4 2017, 10:11 PM

Smalyshev moved this task from Backlog to Next on the User-Smalyshev board.

Smalyshev mentioned this in T125500: [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch.Aug 7 2017, 5:35 PM

Smalyshev moved this task from Next to Waiting/Blocked on the User-Smalyshev board.Aug 25 2017, 4:58 AM

Change 375073 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[mediawiki/extensions/Wikibase@master] Tune Wikidata entity search parameters

https://gerrit.wikimedia.org/r/375073

gerritbot added a project: Patch-For-Review.Aug 31 2017, 8:21 PM

Smalyshev moved this task from Waiting/Blocked to Doing on the User-Smalyshev board.Aug 31 2017, 8:29 PM

It would be very nice if we had a setup that would allow us to compare different search profiles side by side, similar to the way we can now compare sql vs cirrus. Can we choose the search profile via a url parameter?

@daniel it is possible to do it, the URL variable is cirrusRescoreProfile. The profiles need to already be in config, of course. If you have ideas which ones you want, please describe them.

Created page https://www.wikidata.org/wiki/User:Smalyshev_(WMF)/Wikidata_search_test for tracking results and problems with current implementation. Please add examples and comments if necessary.

Smalyshev moved this task from Doing to Waiting/Blocked on the User-Smalyshev board.Sep 6 2017, 7:35 AM

Smalyshev moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.

Change 375073 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Tune Wikidata entity search parameters

https://gerrit.wikimedia.org/r/375073

Smalyshev closed this task as Resolved.Sep 12 2017, 5:05 PM

Smalyshev moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.

Smalyshev mentioned this in T175741: Set ElasticSearch implementation as default for wbsearchentites on Wikidata.Sep 12 2017, 7:59 PM