Add full-text search support to Query Service
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	Smalyshev
	Aug 1 2016, 9:13 PM

Description

Currently, Query Service has limited capability to search through the data, due to absence of support for FTS engine. We did not add this capability, since we already have FTS implementation in ElasticSearch - however, we do not actually use it from Blazegraph.

Blazegraph allows to plug in external search engines: https://wiki.blazegraph.com/wiki/index.php/ExternalFullTextSearch

We should develop a capability to use our search from inside Blazegraph, there are a number of requests for queries that include searching and right now they are not running efficiently due to inability to leverage ElasticSearch.

Current workaround: use SQL on quarry.wmflabs.org

Related Objects
Search...

Status	Assigned	Task
Declined	None	T141813 Add full-text search support to Query Service
Resolved	Smalyshev	T125500 [Epic] Index Wikidata labels and descriptions as separate fields in ElasticSearch
Invalid	None	T132444 [Task] Create repository for WikibaseElastic
Resolved	Smalyshev	T150891 Find a good way to represent multi-lingual text fields in Elastic
Resolved	Smalyshev	T155139 'ContentHandlerForModelID' hook allows creating of handlers that aren't registered
Resolved	Smalyshev	T157604 Make Wikibase register all its content handlers
Resolved	Smalyshev	T157626 ContentHandlerSanityTest assumes every handler can create empty content, but Wikibase can't
Resolved	dcausse	T160926 Make noop script be able replace whole fields with nested subfields
Resolved	dcausse	T166589 Update wikidata code to take advantage of nested fields noop script
Resolved	Smalyshev	T171548 Wikidata Elastic search ignores limit
Resolved	debt	T172422 Add stats marker to wikidata entity search
Resolved	Smalyshev	T172467 Make good prefix search profile for Wikidata entities
Resolved	Smalyshev	T173231 Wikidata Elastic search drops results with matches in different language label
Resolved	Smalyshev	T175741 Set ElasticSearch implementation as default for wbsearchentites on Wikidata
Resolved	Smalyshev	T178851 Use label & description index for fulltext search
Resolved	Smalyshev	T176903 Index wikidata descriptions
Resolved	Smalyshev	T180169 Make list of languages where using stemmed analyzer for Wikibase is beneficial
Resolved	Smalyshev	T182271 Wikidata fulltext search should handle search syntax properly
Resolved	dcausse	T182293 Tune wikidata fulltext search similarity parameters
Resolved	Smalyshev	T181426 Reindex wikidata to enable description index

Event Timeline

Smalyshev created this task.Aug 1 2016, 9:13 PM

Restricted Application added projects: Wikidata, Discovery-ARCHIVED. · View Herald TranscriptAug 1 2016, 9:13 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Smalyshev moved this task from Needs triage to WDQS on the Discovery-ARCHIVED board.Aug 2 2016, 6:45 PM

Smalyshev merged a task: T144280: [feature request] make Wikidata site search available as a service at query.wikidata.org.Aug 30 2016, 4:14 PM

Smalyshev added a subscriber: Esc3300.

Esc3300 added a parent task: T86530: Replace wb_terms table with more specialized mechanisms for terms (tracking).Aug 30 2016, 4:20 PM

Esc3300 updated the task description. (Show Details)

Sample syntax from T144280:

  SERVICE wikibase:sitesearch { 
		?item wdt:P31 ?instance .
      bd:serviceParam wikibase:searchsite "wikidatawiki"  . 
      bd:serviceParam wikibase:searchterm "John Doe"  . 
      bd:serviceParam wikibase:searchlang "en"  . 
      bd:serviceParam wikibase:searchresultlimit "500" . 
      bd:serviceParam wikibase:searchresultoffset "10" . 
  }

Esc3300 awarded a token.Aug 30 2016, 4:22 PM

thiemowmde triaged this task as Low priority.Aug 31 2016, 7:40 AM

thiemowmde added a subscriber: • Jonas.

Smalyshev moved this task from Incoming to Current work on the Wikidata-Query-Service board.Sep 9 2016, 1:12 AM

Smalyshev raised the priority of this task from Low to Medium.Sep 12 2016, 10:46 PM

jberkel subscribed.Oct 27 2016, 8:32 AM

Mushroom subscribed.Nov 3 2016, 5:45 PM

Esc3300 mentioned this in T153623: find an alternative for bulk use of Autolist2.Dec 19 2016, 8:01 AM

JanusTroelsen subscribed.Feb 9 2017, 10:23 PM

Scott_WUaS subscribed.Feb 17 2017, 9:31 PM

Abbe98 subscribed.May 4 2017, 1:07 PM

Joeyiscream subscribed.May 9 2017, 6:01 AM