Page MenuHomePhabricator

Explore search parameters to increase diversity in topic-based suggestions
Closed, ResolvedPublic

Description

As part of the work to support Custom Translation Suggestions (T113257), users can select specific topic areas to get suggestions in their area of interest. However, some of the topic areas such as entertainment and engineering result in suggestions dominated by very specific topics (planes and wrestlers) resulting in a lack of diversity for the results.

The articletopic: keyword on search (documentation) is used for querying articles on a specific topic area, and the underlying model is used for recommendations in tools such as Content Translation. The example results for entertainment and for engineering can be checked when searching English Wikipedia and are illustrated below.

engineeringentertainment
en.wikipedia.org_w_index.php_search=articletopic%3Aengineering&title=Special_Search&profile=advanced&fulltext=1&ns0=1(Wiki Tablet).png (768×1,024 px, 270 KB)
en.wikipedia.org_w_index.php_search=articletopic%3A+entertainment&title=Special_Search&profile=advanced&fulltext=1&ns0=1(Wiki Tablet).png (768×1,024 px, 267 KB)

Based on the input from T376987#10222124, it was suggested that adjusting some of [the Search API parameters such as srsort or srqiprofile could have a randomizing effect on the results that will increase their diversity (while still showing results that the model classified as relevant for the selected topic area). An excerpt from Isaac's comment is shared below, which includes more context:

You could consider changing the "generically more-relevant" part of the sorting to something that exerts a more randomizing effect on the results topic-wise. So here's the docs for the Search API and we could either play with srsort or srqiprofile. The default I think is this srqiprofile=wsum_inclinks_pv&srsort=relevance. I looked and I think Growth is using srqiprofile=classic_noboostlinks (code), which does mix things up. There are options you can play with -- here are the docs for srsort and srqiprofile.

The purpose of this task is to explore and document the different options that the search parameters provide, and which are the final results they produce. Ideally we should identify the approach that produces results that are very relevant to the selected topic, cover different aspects of it and include a minimal number of unrelated topics (false positives).

Event Timeline

Pginer-WMF triaged this task as Medium priority.

The API sandbox is a good tool to try out various API parameters.

As suggested, it looks like gsrqiprofile=classic_noboostlinks calms down the relevance algorithm that brings a small part of the topic always to the top (aircraft in the case of engineering). Also, gsrsort=random brings variety to the results. We have shuffling in place in the recommendation API but gsrsort acts on a bigger dataset being considered by the search engine before results are sent back.

Here's some examples in the table below.

TopicBaselinegsrqiprofile=classic_noboostlinksgsrqiprofile=classic_noboostlinks + gsrsort=random
EngineeringLinkLinkLink
EntertainmentLinkLinkLink

Change #1081237 had a related patch set uploaded (by Sbisson; author: Sbisson):

[research/recommendation-api@master] Tweak srqiprofile and gsrsort for better search results variety

https://gerrit.wikimedia.org/r/1081237

Change #1081237 merged by jenkins-bot:

[research/recommendation-api@master] Tweak gsrqiprofile and gsrsort for better search results variety

https://gerrit.wikimedia.org/r/1081237

Change #1087156 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[research/recommendation-api@master] Use random sorting only for topic-based recommendations

https://gerrit.wikimedia.org/r/1087156

Change #1087156 merged by jenkins-bot:

[research/recommendation-api@master] Use random sorting only for topic-based recommendations

https://gerrit.wikimedia.org/r/1087156

Change #1088276 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update recommendation-api to 2024-11-06-190017-production

https://gerrit.wikimedia.org/r/1088276

Change #1088276 merged by jenkins-bot:

[operations/deployment-charts@master] Update recommendation-api to 2024-11-08-142328-production

https://gerrit.wikimedia.org/r/1088276

Nikerabbit subscribed.

For @Pginer-WMF as the reporter to check.

There has been an improvement. More on "Entertainment" than "Engineering' where airplane-related articles seem to be quite common still.

EngineeringEntertainment
bn.m.wikipedia.org_w_index.php_title=Special_ContentTranslation&filter-type=topic&filter-id=engineering&active-list=suggestions&from=en&to=bn(Wiki Tablet).png (768×1,024 px, 171 KB)
bn.m.wikipedia.org_w_index.php_title=Special_ContentTranslation&filter-type=topic&filter-id=engineering&active-list=suggestions&from=en&to=bn(Wiki Tablet) (1).png (768×1,024 px, 111 KB)

Given the improvements, I think we can close this ticket and consider follow-up experimentation based on data. A/B testing can be a useful tool to identify which possible approaches surface more useful suggestions to users.