Index wikidata descriptions
Closed, ResolvedPublic
Actions

Description

Enable indexing of Wikibase descriptions in wikidata ElasticSearch index.

This probably requires figuring out how to apply language-specific analyzers to specific description fields.

Details

Subject	Repo	Branch	Lines +/-
Move most of the config building to CirrusSearch	mediawiki/extensions/Wikibase	master	+31 -1 K
Port per-language indexing code from Wikibase to CirrusSearch	mediawiki/extensions/CirrusSearch	master	+1 K -0
Add stemming languages settings for description indexing	operations/mediawiki-config	master	+45 -0
Index descriptions for Wikibase entites	mediawiki/extensions/Wikibase	master	+1 K -36

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	• Wikidata-bugs	T77898 query taking 10s: TermSqlIndex::getMatchingIDs
Open	None	T46529 Wikidata search problems (tracking)
Resolved	aude	T110648 [Bug] high-ranking items seemed to have dropped significantly in Special:Search results for wikidata
Resolved	Smalyshev	T119067 Adjust rescoring config for Wikidata to consider sitelink count
Resolved	aude	T119066 Add sitelink count to search index for Wikidata
Resolved	None	T120217 [Task] Add CirrusSearch and Elastica as dependencies of Wikibase on Jenkins
Invalid	None	T161334 Stemming for item suggestions, e.g. "the" vs. no "the" on Wikidata
Resolved	Smalyshev	T78157 [Story] Use ElasticSearch for entity search on wikidata.org
Resolved	Lydia_Pintscher	T88535 [Task] Investigate preconditions & constraints for using Elastic directly
Duplicate	Smalyshev	T117520 Index Wikidata labels, aliases and descriptions as separate fields in ElasticSearch
Duplicate	None	T117522 Create standard "completion suggestion" API for Search
Declined	None	T120089 Add an internal completion or suggestions API to core SearchEngine
Declined	None	T170392 Create gadget that enables the use of the elastic search backend for the entity selector
Resolved	daniel	T170400 Define metrics for search result quality for the entity selector widget on wikidata.
Resolved	Lydia_Pintscher	T170405 Manually evaluate cirrus based entity search on test system
Open	None	T170547 Metrics to evaluate new search for item suggestor
Declined	None	T170549 Provide A/B test for item suggestor
Resolved	Smalyshev	T162292 Reindex wikidata to pick up labels/descriptions mappings
Resolved	Smalyshev	T175199 Index certain statements for Wikidata items
Declined	None	T141813 Add full-text search support to Query Service
Resolved	Smalyshev	T125500 [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch
Resolved	dcausse	T160926 Make noop script be able replace whole fields with nested subfields
Resolved	dcausse	T166589 Update wikidata code to take advantage of nested fields noop script
Resolved	Smalyshev	T178851 Use label & description index for fulltext search
Resolved	Smalyshev	T176903 Index wikidata descriptions
Resolved	Smalyshev	T180169 Make list of languages where using stemmed analyzer for Wikibase is beneficial

Event Timeline

Smalyshev created this task.Sep 27 2017, 7:20 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 27 2017, 7:20 PM

Smalyshev added a parent task: T125500: [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch.Sep 27 2017, 7:20 PM

Smalyshev mentioned this in T125500: [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch.

Change 381068 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[mediawiki/extensions/Wikibase@master] [WIP] Index descriptions for Wikibase

https://gerrit.wikimedia.org/r/381068

gerritbot added a project: Patch-For-Review.Sep 27 2017, 7:27 PM

Probably makes sense to use results of T147959 (once it is done) since now fallbacks don't make much sense and also using them turns out to be rather slow with all 427 languages supported.

debt triaged this task as Medium priority.Oct 3 2017, 5:35 PM

Smalyshev moved this task from Incoming to not in use - please delete on the Discovery-Search (Current work) board.Oct 3 2017, 5:35 PM

TJones mentioned this in T147959: Generic language fallbacks in Mediawiki should not be used for Elasticsearch language analyzers.Oct 3 2017, 7:16 PM

thiemowmde assigned this task to Smalyshev.Oct 10 2017, 9:34 AM

thiemowmde added projects: Wikidata, MediaWiki-extensions-WikibaseRepository, Wikidata-Former-Sprint-Board.

thiemowmde moved this task from incoming to consider for next sprint on the Wikidata board.

thiemowmde moved this task from Proposed to Monitoring on the Wikidata-Former-Sprint-Board board.

thiemowmde added subscribers: Lydia_Pintscher, hoo, aude.

Smalyshev mentioned this in T178851: Use label & description index for fulltext search.Oct 23 2017, 11:19 PM

Smalyshev added a subtask: T178851: Use label & description index for fulltext search.

Smalyshev removed a subtask: T178851: Use label & description index for fulltext search.Oct 23 2017, 11:21 PM

Smalyshev added a parent task: T178851: Use label & description index for fulltext search.

Smalyshev moved this task from not in use - please delete to Needs review on the Discovery-Search (Current work) board.Nov 8 2017, 5:56 PM

Smalyshev created subtask T180169: Make list of languages where using stemmed analyzer for Wikibase is beneficial.Nov 9 2017, 9:23 PM

Smalyshev closed subtask T180169: Make list of languages where using stemmed analyzer for Wikibase is beneficial as Resolved.Nov 9 2017, 11:52 PM

TJones mentioned this in T180365: Remove unneeded language-specific config for lowercase filters.Nov 13 2017, 3:57 PM

thiemowmde added a project: User-thiemowmde.Nov 22 2017, 4:06 PM

thiemowmde subscribed.

thiemowmde moved this task from Incoming to Monitoring on the User-thiemowmde board.Nov 22 2017, 5:12 PM

Change 392894 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/mediawiki-config@master] Add stemming languages settings for description indexing

https://gerrit.wikimedia.org/r/392894

Lydia_Pintscher removed a project: Wikidata-Former-Sprint-Board.Nov 23 2017, 1:35 PM

Change 381068 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Index descriptions for Wikibase entites

https://gerrit.wikimedia.org/r/381068

Smalyshev mentioned this in T181426: Reindex wikidata to enable description index.Nov 27 2017, 6:38 PM

Smalyshev moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.Nov 27 2017, 6:52 PM

Change 392894 merged by jenkins-bot:
[operations/mediawiki-config@master] Add stemming languages settings for description indexing

https://gerrit.wikimedia.org/r/392894

Mentioned in SAL (#wikimedia-operations) [2017-11-27T19:21:09Z] <catrope@tin> Synchronized wmf-config/Wikibase.php: T176903 (duration: 00m 45s)

Change 393689 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[mediawiki/extensions/CirrusSearch@master] [WIP] Port per-language indexing code from Wikibase to CirrusSearch

https://gerrit.wikimedia.org/r/393689

Change 393690 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[mediawiki/extensions/Wikibase@master] [WIP] Move most of the config building to CirrusSearch

https://gerrit.wikimedia.org/r/393690

thiemowmde updated the task description. (Show Details)Nov 28 2017, 10:23 AM

Change 393689 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Port per-language indexing code from Wikibase to CirrusSearch

https://gerrit.wikimedia.org/r/393689

Change 393690 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Move most of the config building to CirrusSearch

https://gerrit.wikimedia.org/r/393690

ReleaseTaggerBot added a project: MW-1.31-release-notes (WMF-deploy-2017-12-05 (1.31.0-wmf.11)).Nov 28 2017, 11:00 PM

Smalyshev closed this task as Resolved.Nov 29 2017, 12:46 AM

thiemowmde removed projects: User-thiemowmde, Patch-For-Review.Nov 29 2017, 9:28 AM

thiemowmde updated the task description. (Show Details)

thiemowmde added a project: Wikidata-Sprint-2017-11-22.Dec 5 2017, 6:32 PM

thiemowmde moved this task from Backlog to Done on the Wikidata-Sprint-2017-11-22 board.