Page MenuHomePhabricator
Create Task
Maniphest T125500

[Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch
Closed, ResolvedPublic
Actions

Assigned To
Smalyshev
Authored By
Tobi_WMDE_SW
Feb 2 2016, 11:09 AM
Tags
Referenced Files
None
Subscribers
Aklapper
aude
daniel
dcausse
gerritbot
hoo
JanZerebecki
View All 21 Subscribers
Tokens
"Like" token, awarded by Liuxinyu970226."Love" token, awarded by Pintoch."Like" token, awarded by Jonas.

Description

This task is for adding labels & descriptions into ElasticSearch and enabling prefix search to use them. Current progress plan:

  • Implement the code for creating & indexing label fields
  • Reindex testwiki and check that index looks sane
  • Reindex wikidata and check that index looks sane (T162292)
  • Add code that allows wbsearchentities to use search engine depending on query flag
  • Setup test page comparing two searches and make an announcement on the list to gather user feedback.
  • Collect feedback and bikeshed about search profiles, weights and result ranking, hopefully arriving to some workable weights profile. (T172467)
  • Set up the config above in production and enable CirrusSearch on wbsearchentities by default (T175741)
  • Discuss & resolve question of how to display entity & title search together
  • Refactor code more to allow opensearch and other code using completionSearch() use the same code as wbsearchentities, in service of the results of the discussion above.
  • Implement the GUI part of the two items above.
  • Figure out how to properly index descriptions (T176903)
  • Make code to allow fulltext search to use entity search when appropriate (T178851).
  • Enable Cirrus searching for Special:Search

More detailed plan: https://www.wikidata.org/wiki/User:Smalyshev_(WMF)/Wikidata_search

Details

SubjectRepoBranchLines +/-
Introduce FieldDefinitions for searchmediawiki/extensions/Wikibasemaster+1 K -480
Enable Cirrus search of wbsearchentities when using useCirrus=1operations/mediawiki-configmaster+3 -0
Add descriptions to ES indexmediawiki/extensions/Wikibasemaster+155 -90
Add prefix search for labelsmediawiki/extensions/Wikibasemaster+2 K -283
Create mappings for Wikibase labelsmediawiki/extensions/Wikibasemaster+374 -26
Generalize field definitions for Items and Propertiesmediawiki/extensions/Wikibasemaster+348 -73
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• Wikidata-bugsT77898 query taking 10s: TermSqlIndex::getMatchingIDs
 OpenNoneT46529 Wikidata search problems (tracking)
 ResolvedaudeT110648 [Bug] high-ranking items seemed to have dropped significantly in Special:Search results for wikidata
 ResolvedSmalyshevT119067 Adjust rescoring config for Wikidata to consider sitelink count
 ResolvedaudeT119066 Add sitelink count to search index for Wikidata
 ResolvedNoneT120217 [Task] Add CirrusSearch and Elastica as dependencies of Wikibase on Jenkins
 InvalidNoneT161334 Stemming for item suggestions, e.g. "the" vs. no "the" on Wikidata
 ResolvedSmalyshevT78157 [Story] Use ElasticSearch for entity search on wikidata.org
 ResolvedLydia_PintscherT88535 [Task] Investigate preconditions & constraints for using Elastic directly
 DuplicateSmalyshevT117520 Index Wikidata labels, aliases and descriptions as separate fields in ElasticSearch
 DuplicateNoneT117522 Create standard "completion suggestion" API for Search
 DeclinedNoneT120089 Add an internal completion or suggestions API to core SearchEngine
 DeclinedNoneT170392 Create gadget that enables the use of the elastic search backend for the entity selector
 ResolveddanielT170400 Define metrics for search result quality for the entity selector widget on wikidata.
 ResolvedLydia_PintscherT170405 Manually evaluate cirrus based entity search on test system
 OpenNoneT170547 Metrics to evaluate new search for item suggestor
 DeclinedNoneT170549 Provide A/B test for item suggestor
ResolvedSmalyshevT162292 Reindex wikidata to pick up labels/descriptions mappings
 ResolvedSmalyshevT175199 Index certain statements for Wikidata items
 DeclinedNoneT141813 Add full-text search support to Query Service
 ResolvedSmalyshevT125500 [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch
 InvalidNoneT132444 [Task] Create repository for WikibaseElastic
 ResolvedSmalyshevT150891 Find a good way to represent multi-lingual text fields in Elastic
 ResolvedSmalyshevT155139 'ContentHandlerForModelID' hook allows creating of handlers that aren't registered
 ResolvedSmalyshevT157604 Make Wikibase register all its content handlers
ResolvedSmalyshevT157626 ContentHandlerSanityTest assumes every handler can create empty content, but Wikibase can't
 ResolveddcausseT160926 Make noop script be able replace whole fields with nested subfields
 ResolveddcausseT166589 Update wikidata code to take advantage of nested fields noop script
 ResolvedSmalyshevT171548 Wikidata Elastic search ignores limit
 ResolveddebtT172422 Add stats marker to wikidata entity search
 ResolvedSmalyshevT172467 Make good prefix search profile for Wikidata entities
 ResolvedSmalyshevT173231 Wikidata Elastic search drops results with matches in different language label
 ResolvedSmalyshevT175741 Set ElasticSearch implementation as default for wbsearchentites on Wikidata
 ResolvedSmalyshevT178851 Use label & description index for fulltext search
 ResolvedSmalyshevT176903 Index wikidata descriptions
 ResolvedSmalyshevT180169 Make list of languages where using stemmed analyzer for Wikibase is beneficial
 ResolvedSmalyshevT182271 Wikidata fulltext search should handle search syntax properly
 ResolveddcausseT182293 Tune wikidata fulltext search similarity parameters
 ResolvedSmalyshevT181426 Reindex wikidata to enable description index

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Smalyshev merged a task: T117520: Index Wikidata labels, aliases and descriptions as separate fields in ElasticSearch.Feb 8 2017, 8:04 AM
Smalyshev mentioned this in T117520: Index Wikidata labels, aliases and descriptions as separate fields in ElasticSearch.
Smalyshev closed subtask T155139: 'ContentHandlerForModelID' hook allows creating of handlers that aren't registered as Resolved.Feb 8 2017, 7:55 PM
gerritbot added a comment.Feb 9 2017, 9:54 PM

Change 336911 had a related patch set uploaded (by Smalyshev):
[WIP] Add indexing for descriptions

https://gerrit.wikimedia.org/r/336911

Tobi_WMDE_SW closed subtask T132444: [Task] Create repository for WikibaseElastic as Invalid.Feb 10 2017, 2:59 PM
• Deskana closed subtask T150891: Find a good way to represent multi-lingual text fields in Elastic as Resolved.Feb 10 2017, 5:26 PM
Smalyshev moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Feb 16 2017, 6:34 AM
• Deskana added a comment.Mar 14 2017, 5:22 PM

Waiting on review from Wikidata folks.

gerritbot added a comment.Mar 20 2017, 12:38 AM

Change 334194 had a related patch set uploaded (by Smalyshev):
[mediawiki/extensions/Wikibase] Create mappings for Wikibase labels

https://gerrit.wikimedia.org/r/334194

Smalyshev mentioned this in T160926: Make noop script be able replace whole fields with nested subfields.Mar 20 2017, 5:22 PM
Smalyshev added a subtask: T160926: Make noop script be able replace whole fields with nested subfields.
thiemowmde added a parent task: T161334: Stemming for item suggestions, e.g. "the" vs. no "the" on Wikidata.Mar 26 2017, 11:46 AM
gerritbot added a comment.Mar 28 2017, 10:56 PM

Change 334019 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Generalize field definitions for Items and Properties

https://gerrit.wikimedia.org/r/334019

gerritbot added a comment.Apr 6 2017, 5:12 PM

Change 334194 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Create mappings for Wikibase labels

https://gerrit.wikimedia.org/r/334194

gerritbot added a comment.Apr 12 2017, 10:37 PM

Change 335728 had a related patch set uploaded (by Smalyshev):
[mediawiki/extensions/Wikibase@master] Add prefix search for labels

https://gerrit.wikimedia.org/r/335728

daniel mentioned this in T163551: Huge number of duplicate rows in wb_terms.Apr 28 2017, 7:03 PM
Smalyshev updated the task description. (Show Details)May 2 2017, 1:14 AM
Smalyshev moved this task from Needs review to not in use - please delete on the Discovery-Search (Current work) board.
Smalyshev added a subscriber: dcausse.
debt closed subtask T160926: Make noop script be able replace whole fields with nested subfields as Resolved.May 30 2017, 5:27 PM
Sjoerddebruin subscribed.Jun 20 2017, 5:44 PM
daniel added a comment.EditedJun 29 2017, 5:56 PM

Issues found while testing on http://elastic-wikidata.wmflabs.org:

  1. Testing on http://elastic-wikidata.wmflabs.org shows that there is a minor issue: When an alias is matched, it is correctly shown in addition to the preferred label. However, if a label in a fallback language is matched (e.g. a search in de-ch matches the en label), that matched label is not shown in addition to the preferred (perhaps de) label. Example: on http://elastic-wikidata.wmflabs.org/index.php/ElasticRepo:Main_Page?uselang=de-ch, enter "H." into the quick search box. This will suggest Q5628592, because the English label is "H. T. Kung" - but it does not show that "H. T. Kung" was matched.
  2. Another issue Katie found when playing with elastic-wikidata.wmflabs.org: We'll need a way to boost exact matches vs. partial matches, or a way to boost label matches vs alias matches. Ideally, we would be able to tweak both. Example: "Poppy" brings up George H.W. Bush as the first match.

Regarding issue (2): if I read EntitySearchHelper correctly, the old system does not boost labels vs. aliases, it treats them exactly the same. It does however strongly prefer exact matches over partial (prefix) matches.

gerritbot added a comment.Jun 29 2017, 10:47 PM

Change 335728 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add prefix search for labels

https://gerrit.wikimedia.org/r/335728

gerritbot added a comment.Jun 29 2017, 10:47 PM

Change 336911 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add descriptions to ES index

https://gerrit.wikimedia.org/r/336911

Smalyshev updated the task description. (Show Details)Jul 3 2017, 8:50 PM
Smalyshev added a comment.EditedJul 3 2017, 10:42 PM

I checked the problem in 1. and it looks like the issue is that the match - at least as reported by elastic - is against labels.de.prefix: "Hsiang-Tsung Kung". This is what ES actually matched - it prefers de fallback with inexact prefix to en fallback with exact one. We could play with weights to make it prefer the more exact match - maybe make discount for using fallback smaller or preference for using exact match bigger. Probably would need help from @dcausse on this, e.g. this query:

http://elastic-wikidata.wmflabs.org/api.php?action=wbsearchentities&search=H.%20T.&format=json&language=de-ch&uselang=de-ch&type=item&cirrusDumpQuery=true

produces this result:

http://elastic-wikidata.wmflabs.org/api.php?action=wbsearchentities&search=H.%20T.&format=json&language=de-ch&uselang=de-ch&type=item&cirrusDumpResult=true

and I'm not sure why exactly.

dcausse added a comment.Jul 6 2017, 6:52 PM

The query seems to use an inexistent field labels_all.near_match changing to labels_all.near_match_folded will fix the highlighting issue.

Smalyshev added a comment.Jul 6 2017, 7:33 PM

Bugfix in: https://gerrit.wikimedia.org/r/#/c/363655/

daniel added a subscriber: Ladsgroup.Jul 8 2017, 11:45 AM

Thanks for the fix @Smalyshev!

@Ladsgroup can you update the elastic test site so it uses the latest master branch, so we can test this? Thanks!

Ladsgroup added a comment.Jul 8 2017, 4:11 PM

@daniel: Done. I also imported around 500 events (chemical elements are too heavy to import)

Smalyshev updated the task description. (Show Details)Jul 18 2017, 7:29 PM
gerritbot added a comment.Jul 21 2017, 5:43 AM

Change 366788 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/mediawiki-config@master] Enable Cirrus search of wbsearchentities when using useCirrus=1

https://gerrit.wikimedia.org/r/366788

Smalyshev updated the task description. (Show Details)Jul 21 2017, 4:54 PM
Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptJul 21 2017, 4:54 PM
gerritbot added a comment.Jul 24 2017, 6:13 PM

Change 366788 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable Cirrus search of wbsearchentities when using useCirrus=1

https://gerrit.wikimedia.org/r/366788

Stashbot subscribed.Jul 24 2017, 6:16 PM

Mentioned in SAL (#wikimedia-operations) [2017-07-24T18:16:40Z] <reedy@tin> Synchronized wmf-config/Wikibase.php: T125500 (duration: 00m 43s)

daniel added subscribers: Lea_Lacroix_WMDE, Lydia_Pintscher.Jul 24 2017, 6:38 PM

@Lea_Lacroix_WMDE @Lydia_Pintscher: this can now be tested live using the (undocumented) useCirrus parameter. Compare:

This doesn't work with the API sandbox, because useCirrus isn't an official parameter of the API module. Perhaps that should be fixed.

Anyway, we can now run a few queries, put the results on wiki pages, and discuss them with the community. Making such pages from the API responses is a "fun" job. Any volunteers?

Smalyshev added a comment.Jul 24 2017, 6:51 PM

I plan to make a page that allows to search with both and compare. We did something like that with completion suggester, I think it shouldn't be too hard to modify it.

Pintoch awarded a token.Jul 24 2017, 7:21 PM
Nikki subscribed.Jul 24 2017, 8:16 PM
Smalyshev added a comment.Jul 24 2017, 11:12 PM

I've made a page for search comparison: http://elastic-wikidata.wmflabs.org/wb.html
(despite it being hosted on elastic-wikidata, the data comes from www.wikidata.org).

Note that descriptions for Elastic search are now broken, due to T162292, as soon as that is done descriptions should be fine.

Smalyshev created subtask T171548: Wikidata Elastic search ignores limit.Jul 24 2017, 11:35 PM
Sjoerddebruin added a comment.Jul 25 2017, 1:24 PM
In T125500#3468439, @Smalyshev wrote:

I've made a page for search comparison: http://elastic-wikidata.wmflabs.org/wb.html
(despite it being hosted on elastic-wikidata, the data comes from www.wikidata.org).

Note that descriptions for Elastic search are now broken, due to T162292, as soon as that is done descriptions should be fine.

Thanks! I've run some first test and I think there are still improvements needed in the field of abbrevations and disambiguation pages.

Sjoerddebruin added a comment.Jul 25 2017, 4:39 PM

I'm keeping track of my experiences on https://www.wikidata.org/wiki/User:Sjoerddebruin/Cirrus now, it seems that not all exact matches are always showed.

Smalyshev updated the task description. (Show Details)Jul 25 2017, 5:35 PM
Smalyshev updated the task description. (Show Details)Jul 25 2017, 9:00 PM
Smalyshev closed subtask T171548: Wikidata Elastic search ignores limit as Resolved.Jul 26 2017, 6:45 PM
debt renamed this task from Index Wikidata labels and descriptions as separate fields in ElasticSearch to [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch.Aug 1 2017, 5:19 PM
debt added a project: Epic.
Smalyshev added a subtask: T172422: Add stats marker to wikidata entity search.Aug 3 2017, 6:32 PM
Smalyshev added a subtask: T172467: Make good prefix search profile for Wikidata entities.Aug 4 2017, 12:33 AM
Smalyshev updated the task description. (Show Details)Aug 7 2017, 5:35 PM
Smalyshev added a subtask: T173231: Wikidata Elastic search drops results with matches in different language label.Aug 12 2017, 10:50 PM
debt closed subtask T172422: Add stats marker to wikidata entity search as Resolved.Aug 14 2017, 6:50 PM
Smalyshev closed subtask T172467: Make good prefix search profile for Wikidata entities as Resolved.Sep 12 2017, 5:05 PM
Smalyshev updated the task description. (Show Details)Sep 12 2017, 7:54 PM
Smalyshev added a subtask: T175741: Set ElasticSearch implementation as default for wbsearchentites on Wikidata.Sep 12 2017, 8:00 PM
Smalyshev updated the task description. (Show Details)Sep 12 2017, 8:03 PM
debt closed subtask T173231: Wikidata Elastic search drops results with matches in different language label as Resolved.Sep 15 2017, 8:20 PM
K4-713 subscribed.Sep 15 2017, 11:44 PM
Smalyshev added a subtask: T176903: Index wikidata descriptions.Sep 27 2017, 7:20 PM
Smalyshev updated the task description. (Show Details)
Smalyshev updated the task description. (Show Details)Oct 23 2017, 11:22 PM
Smalyshev added a subtask: T178851: Use label & description index for fulltext search.
Smalyshev closed subtask T175741: Set ElasticSearch implementation as default for wbsearchentites on Wikidata as Resolved.Oct 25 2017, 6:27 PM
thiemowmde mentioned this in T179061: pasting full URL into entity selector no longer works.Oct 26 2017, 10:46 AM
Smalyshev updated the task description. (Show Details)Nov 2 2017, 10:25 PM
Smalyshev moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Nov 14 2017, 6:14 PM
Smalyshev added a subtask: T181426: Reindex wikidata to enable description index.Nov 27 2017, 6:38 PM
• Deskana unsubscribed.Nov 28 2017, 12:49 PM
Smalyshev closed subtask T176903: Index wikidata descriptions as Resolved.Nov 29 2017, 12:46 AM
Smalyshev moved this task from Needs review to not in use - please delete on the Discovery-Search (Current work) board.Nov 29 2017, 6:32 PM
Liuxinyu970226 awarded a token.Nov 30 2017, 11:17 AM
Liuxinyu970226 subscribed.
Smalyshev closed subtask T181426: Reindex wikidata to enable description index as Resolved.Dec 1 2017, 9:42 PM
Smalyshev updated the task description. (Show Details)
Lydia_Pintscher moved this task from ready to go to in progress on the Wikidata board.Dec 18 2017, 2:56 PM
Smalyshev moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Mar 9 2018, 8:59 PM
Smalyshev moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Mar 16 2018, 8:01 PM
Smalyshev updated the task description. (Show Details)Mar 22 2018, 8:45 PM
Smalyshev added a comment.Mar 22 2018, 8:55 PM

I've moved the "combined search" part of this ticket to T190454, and I think the rest for it is done, so I'll resolve it soon unless there are objections.

Smalyshev closed this task as Resolved.Apr 3 2018, 12:52 AM
Smalyshev closed subtask T178851: Use label & description index for fulltext search as Resolved.

I am resolving this as "combined search" part now has its own task.

Liuxinyu970226 unsubscribed.Apr 3 2018, 12:48 PM
Liuxinyu970226 subscribed.Apr 4 2018, 4:59 AM
Liuxinyu970226 unsubscribed.
hoo added a parent task: T141813: Add full-text search support to Query Service.Apr 12 2018, 12:47 PM
gerritbot added a comment.Jul 27 2018, 11:07 PM

Change 287161 abandoned by Aude:
Introduce FieldDefinitions for search

https://gerrit.wikimedia.org/r/287161

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL