Page MenuHomePhabricator

Test articlecountry: search results for relevance with the current RecAPI
Closed, ResolvedPublic

Description

Following the completion of T386068: Implement articlecountry a new CirrusSearch keyword, the articlecountry: keyword on search can now be used for querying articles for a specific country. This CirrusSearch feature works by searching with keyword articlecountry: + a coutry code (according to the list of ISO 3166 country codes). You can see example of the raw results here:

The LPL team would like to test and determine if this new feature can be used to offer appropriate translation suggestions, based on how our current CX RecAPI T367873 works.

Event Timeline

PWaigi-WMF moved this task from Backlog to Prioritized on the LPL Hypothesis board.

Just flagging T377124 as similar in case there are any insights there that are helpful.

The raw results provided directly by the articlecountry: feature are a good starting point for our recommendation API to provide appropriate translation suggestions. On a first look, the raw results can seem a bit odd:

Raw results for "Switzerland": articlecountry:che (link)
Albert Einstein (...German-born theoretical physicist)
Alps (...extensive mountain ranges in Europe)
James Joyce (...Irish novelist, poet, and literary critic)
Antarctic Treaty System (...agreements to regulate international relations)
Alemanni (...confederation of Germanic tribes)

Yet those raw results from articlecountry: are accurate in the sense that the evaluation criteria for the model itself, as I understand, is based on accuracy: the results have something to do with the country in query. The raw results we get with a basic query are sorted by popularity by default, so we see popular articles with a minor connection to the country populating the top results.

In order to get a more appropriate set of recommendations, there are a couple of actions we can take. First in our recommendation API we can query with gsrqiprofile=classic_noboostlinks to get results not sorted by popularity as this gives us more general results that are closer to what we expect (popular articles can add some distortion to our results). Likewise we can pair with gsrqiprofile=classic_noboostlinks with gsrsort=random to randomize results.

Raw results for "Switzerland": articlecountry:che (link)Results for "Switzerland": articlecountry:che&gsrqiprofile=classic_noboostlinks&gsrsort=random (link)
Albert Einstein (...German-born theoretical physicist)Solothurn (...capital of the canton of Solothurn in Switzerland)
Alps (...extensive mountain ranges in Europe)Hans Schaffner (...Swiss politician and member of the Swiss Federal Council)
James Joyce (...Irish novelist, poet, and literary critic)Walther Stampfli (...Swiss politician and member of the Swiss Federal Council )
Antarctic Treaty System (...agreements to regulate international relations)Raetia (...province of the Roman Empire that occupied eastern and central Switzerland)
Alemanni (...confederation of Germanic tribes)Richard Scarry (...American children's author and illustrator )

In addition, we could also do an extra query for a set of results to look at the weighted_tags ( as offered by this internal module). The weight value provided goes from 0 to 1000, so looking at these values we can choose items that are closer to 1000 to increase our confidence on of the recommendations. See the same results from table above with weighted_tag value for more context:

Raw results for "Switzerland": articlecountry:che (link)Results for "Switzerland": articlecountry:che&gsrqiprofile=classic_noboostlinks&gsrsort=random (link)
Albert Einstein (666/1000)Solothurn (1000/1000)
Alps (1000/1000)Hans Schaffner (1000/1000)
James Joyce (761/1000)Walther Stampfli (1000/1000)
Antarctic Treaty System (1000/1000)Raetia (1000/1000)
Alemanni (1000/1000)Richard Scarry (500/1000)

We could discard Albert Einstein or James Joyce or Richard Scarry, for example, in favor of ones with higher scores. This additional query would add an extra cost which would have to be considered during implementation to understand if it's really worth it.

All in all, it seems that querying with gsrqiprofile=classic_noboostlinks&gsrsort=random and filtering out items with lower weighted tag values can allow the recommendation API to provide appropriate suggestions.

A quick note on the cirrusdoc API prop, it's a debugging interface for the search team and should not be used in a production setting.

Thanks Stephane, indeed it's stated as such in the documentation. I believe even without the weighted_tags perspective we still have enough for T391230. If we all agree this is the case, we can move this ticket (T388942) to the next logical column