Change Details

It appears that all textual properties are being removed from the indexed text search content. As far as I understand `repo/config/Wikibase.searchindex.php` is responsible for this and it looks like it stopped to be registered as a `WikibaseTextForSearchIndex` hook. [[https://www.wikidata.org/wiki/Wikidata:Contact_the_development_team#Search_doesn't_include_subtitle_field?|Original report]]

In T163642 we made all strings of `index statements` part of the all field allowing them to be searchable by plain search queries. Unfortunable only a subset of the statements are being indexed. Reason is that indexing statements today means that we populate the statement_keyword field. This is something we do not want to do because textual content (phrases, long text that needs tokenization) is not suited for keyword matching. So if we want to increase recall on wikidata using textual we need to come up with a new solution to populate extra text content to existing CirrusSearch field. Currently the text fields are: - text: populated using \Wikibase\EntityContent::getTextForSearchIndex - auxiliary_text: not used by EntityHandler We should evaluate the impact on the size of the index to know if can feed all the textual properties or only a subset.

It appears thatIn T163642 we made all textual properties are being removed fromstrings of `index statements` part of the all field allowing the indexed textm to be searchable by plain search contentqueries. As far as I understand `repo/config/Wikibase.searchindex.php` is responsible for this and it looks like it stopped to be registered as a `WikibaseTextForSearchIndex` hookUnfortunable only a subset of the statements are being indexed. Reason is that indexing statements today means that we populate the statement_keyword field. This is something we do not want to do because textual content (phrases, long text that needs tokenization) is not suited for keyword matching. So if we want to increase recall on wikidata using textual we need to come up with a new solution to populate extra text content to existing CirrusSearch field. Currently the text fields are: - text: populated using \Wikibase\EntityContent::getTextForSearchIndex - auxiliary_text: not used by EntityHandler We should evaluate the impact on the size of the index to know if can feed all the textual properties or only a subset. [[https://www.wikidata.org/wiki/Wikidata:Contact_the_development_team#Search_doesn't_include_subtitle_field?|Original report]]