Change Details

In order to store Wikibase terms, such as labels and aliases, in Elastic, we need to find a good way to represent multi-lingual values. That representation has to support language fallback: If de-ch falls be to de, then searching for de-ch:Haus should also find de:Haus (and possibly also vice versa). Implementation ideas: Language fallback support can be achieved using index expansion (indexing de:Haus also as de-ch:Haus) or query expansion (a search for de-ch:Haus turns into a search for de-ch:Haus or de:Haus). Index expansion requires more space, and query expansion requires more time. A compromise could be a multi-value "all languages" field in addition to the per-language fields. This would make it possible to implement language fallback programmatically, without greatly increasing storage size and schema complexity. For instance: If there is only an english label, and all languages fall back to english, and we have 100 languages configured, index expansion would store the english label 100 times. The all-languages approach would store it twice. However, all-languages needs two queries (one for the exact match, and one for all-languages), and the second can potentially have a large result set to process. Simple query expansion also rarely needs more than two queries

In order to store Wikibase terms, such as labels and aliases, in Elastic, we need to find a good way to represent multi-lingual values. That representation has to support language fallback: If de-ch falls be to de, then searching for de-ch:Haus should also find de:Haus (and possibly also vice versa). Note that the term representation in Elastic is not merely intended a search index, but also for retrieving all labels/descriptions for a given subject. Implementation ideas: Language fallback support can be achieved using index expansion (indexing de:Haus also as de-ch:Haus) or query expansion (a search for de-ch:Haus turns into a search for de-ch:Haus or de:Haus). Index expansion requires more space, and query expansion requires more time. A compromise could be a multi-value "all languages" field in addition to the per-language fields. This would make it possible to implement language fallback programmatically, without greatly increasing storage size and schema complexity. For instance: If there is only an english label, and all languages fall back to english, and we have 100 languages configured, index expansion would store the english label 100 times. The all-languages approach would store it twice. However, all-languages needs two queries (one for the exact match, and one for all-languages), and the second can potentially have a large result set to process. Simple query expansion also rarely needs more than two queries. However, all-languages provides a cheap way to get all labels in all languages.

In order to store Wikibase terms, such as labels and aliases, in Elastic, we need to find a good way to represent multi-lingual values. That representation has to support language fallback: If de-ch falls be to de, then searching for de-ch:Haus should also find de:Haus (and possibly also vice versa). Note that the term representation in Elastic is not merely intended a search index, but also for retrieving all labels/descriptions for a given subject. Implementation ideas: Language fallback support can be achieved using index expansion (indexing de:Haus also as de-ch:Haus) or query expansion (a search for de-ch:Haus turns into a search for de-ch:Haus or de:Haus). Index expansion requires more space, and query expansion requires more time. A compromise could be a multi-value "all languages" field in addition to the per-language fields. This would make it possible to implement language fallback programmatically, without greatly increasing storage size and schema complexity. For instance: If there is only an english label, and all languages fall back to english, and we have 100 languages configured, index expansion would store the english label 100 times. The all-languages approach would store it twice. However, all-languages needs two queries (one for the exact match, and one for all-languages), and the second can potentially have a large result set to process. Simple query expansion also rarely needs more than two queries. However, all-languages provides a cheap way to get all labels in all languages.