Page MenuHomePhabricator

[A/B Test] Add 'explore similar' pages, categories and suggested languages in search results
Closed, ResolvedPublic

Description

Based on the finalized design from T146663 and T139310, this A/B test will have a test group that will be shown additional information that will be relevant to individual results on the existing search results page. This additional information will be links and metadata of related pages with links to related categories and suggested language links that are similar to the individual search results. A control group will see the currently existing search results page.

This test is expected to last a week and will probably be run on English Wikipedia (desktop only) in June 2017. We'll use this ticket for the gerrit code check-ins and for turning the test on and off.

Test group users will see for each search result returned:

  • 'explore similar' link and dropdown icon
  • when the user hovers over the link of the search result, we will roll down a display below the result:
  • a label 'Related pages' and
    • up to 3 related page links containing metadata (as available):
      • thumbnail image
      • name of related pages
      • metadata description
  • a label 'Categories' and
    • a 3 column listing of links
      • up to 3 links of related categories per column (for a maximum total display of 9 related categories)
  • a label 'Suggested languages'
    • display of additional languages that the article is available in
    • this uses the preferred browser language settings and the regional languages (using the universal language selector - user's IP region, browser preferred lang and the lang that the user has set as preferred within the Wikipedia account UI)
  • when the user moves away from the new box, the links will roll back up
  • if there aren't any articles in different languages, message that and add a link to translatewiki

Bucketing

Bucket testing logic generally is as follows:

  • 1 in 1000 users are included in EventLogging (on enwiki that's about 2K full-text searchers, according to T163273)
  • Of those 1 in 1000 users, 1 in 2 are included in the test
  • Of those 1 in 2 users
    • 50% go in a test group, labeled "serp_explore_similar"
    • the other 50% of users will go in a control group, labeled "serp_no_similar_links"
  • The remaining chunk (~1K) of the original bucketed ~2K users will get a NULL (the string null, or the MySQL null, we can detect either).

EventLogging

Eventlogging needs to capture:

  • if the user hovered over the 'explore' links (to open the listings)
  • if results for related pages were shown and how many
  • if results for related categories were shown and how many
  • if results for languages were shown and how many
  • if the user clicked on a related page and which one
  • if the user clicked on a related category and which one
  • if the user clicked on a language link and which one
  • we'd like to know if overall clicks go up or down for the search results





Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
debt assigned this task to Jdrewniak.Feb 8 2017, 5:04 PM

A static prototype of the 'explore similar' feature is available at this URL
https://people.wikimedia.org/~jdrewniak/SERP-sidebar/index.htm

debt added a comment.Feb 24 2017, 1:57 AM

Current view of this prototyped widget is this:

debt added a comment.Mar 1 2017, 4:30 PM

FYI - anyone can test out the 'explore similar' and wiktionary widgets by adding these lines to your common.js file on enwiki:

importScript( 'User:JDrewniak_(WMF)/exploreSimilarSearchResults.js' ); Backlink: [[User:JDrewniak_(WMF)/exploreSimilarSearchResults.js]]
importScript( 'User:JDrewniak_(WMF)/wiktionaryDefinitionInSearchResults.js' );
Backlink: [[User:JDrewniak_(WMF)/wiktionaryDefinitionInSearchResults.js]]

debt added a comment.Mar 1 2017, 5:03 PM

Hi @Jdrewniak - here are some recommended updates to the widgets from recent feedback:

Recommended updates to the wiktionary widget:

  • add link to the actual definition shown by using the 'Definition' word as the link
  • add link to a search results page on wiktionary.org for all results for the definition, using the 'wiktionary.org' word as the link
  • wiktionary.org experiemental API might need to be updated to:
    • display the obsolete flag so that we can disregard that entry
    • display the most common definition
  • when displayed with the sister project search results, don't have the sister project results pushed down the page on the right hand side:

Recommended updates to the explore similar widget:

  • down arrow icon should switch to up arrow icon when the explore similar results are displayed
  • display default missing icon image if the related articles that are shown do not images to display
debt added a comment.Mar 2 2017, 6:49 PM

Also, there is a known issue where the wiktionary widget displays (oftentimes) a different definition than what we show in the sister project search results. This is because the experimental API for the wiktionary widget displays the first result it finds and the sister project search results displays the top ranked/used definition. Here's a screenshot of an example:

EBernhardson added a comment.EditedMar 7 2017, 6:37 PM

I think it will be reasonable to integrate the related articles code into CirrusSearch. The javascript code would probably attach to the existing ext.cirrus.serp module that we ship to search result pages. I suppose as it depends on it though, the RelatedArticles extension might not be the worst place to put it either (using the same php hooks as ext.cirrus.serp to be added to the search result page js).

I'm not sure about the wiktionary one, it is very specific. Need to look for some precedent on how it's been done previously.

debt added a comment.Mar 7 2017, 8:23 PM

Thanks, @EBernhardson for the info! We might need to make the wiktionary widget into a Beta feature for testing purposes, since it's only available on enwiki.

debt added a comment.Mar 14 2017, 4:57 PM

I've added ticket T160434 to address the updates for the wiktionary widget - in order to get it ready for a beta test on enwiki.

@Jdrewniak said this is on the backburner a little bit due to the results from the first A/B test. He thinks this is still in progress.

debt added a comment.Apr 17 2017, 6:46 PM

I've created a test account on enwiki for usertesting.com and testing will begin in the next week or so; we'll commence an A/B test soon as well.

debt renamed this task from [A/B Test] Add 'explore similar' pages and categories for search results to [A/B Test] Add 'explore similar' pages, categories and suggested languages in search results.May 9 2017, 4:26 PM
debt updated the task description. (Show Details)
mpopov updated the task description. (Show Details)May 9 2017, 5:58 PM
mpopov added a comment.May 9 2017, 6:59 PM

@Jdrewniak & @EBernhardson & @chelsyx:

Eventlogging needs to capture:

  • if the user hovered over the 'explore similar' link (to open the listings)
    • this will also capture if the user clicked on the link
  • if the user hovered over the same 'explore similar' link (to close the listings)
    • this will also capture if the user clicked on the link

I suggest tracking this via 'explore-similar-on' & 'explore-similar-off' in the action enum

I'm also thinking that we should generate a random exploreSimilarID to use for explicit pairing the on & off events, rather than making assumptions in post-hoc session reconstruction via timestamps. This would go into the extraParams field of 'explore-similar-*' events.

  • if results for related pages were shown and how many

I suggest adding a "related": ## to explore-similar-on events' extraParams

  • if results for related categories were shown and how many

I suggest adding a "categories": ## to explore-similar-on events' extraParams

  • if the user clicked on a related page and which one

I suggest tracking this via 'esclick' in the action enum. These events would still have a position recorded like regular 'click' events for tracking if the user clicked on a related page for the N-th result on the page overall. We record which related page the user clicked on as 'relatedPosition' in 'esclick' events' extraParams field. The destination URL is recorded as a 'destination' in extraParams also similar to how we did in the cross-wiki A/B test and how we do on the Wikipedia.org portal eventlogging.

  • if the user clicked on a related category and which one

Same as related position but 'categoryPosition'

  • if the user clicked on a language link and which one

Same as related position but 'languagePosition'

I'm also thinking that we should generate a random exploreSimilarID to use for explicit pairing the on & off events, rather than making assumptions in post-hoc session reconstruction via timestamps.

Can you elaborate on that? What assumptions would we make?

I'm also thinking that we should generate a random exploreSimilarID to use for explicit pairing the on & off events, rather than making assumptions in post-hoc session reconstruction via timestamps.

Can you elaborate on that? What assumptions would we make?

Sure! Okay, so the assumption that I think we would make is that every explore-similar-on event after the initial one would be separated by an explore-similar-off event when we order the events by timestamp.

That is, when we look at the sequence of events we should only see something like { on, off, on, off, on, click, off, on, off, on, click, off }, but because technical things can sometimes go wrong, I don't think it's impossible for us to observe a nonsensical sequence like { on, on, off, on, click, on, off, off, click } in the wild. So an identifier that would let us explicitly and accurately link on/click/off events together could act as a safeguard against potential problems.

I also accept the possibility that I'm being too paranoid/cynical :)

I see. That make sense. Thank you @mpopov !

I also accept the possibility that I'm being too paranoid/cynical :)

I don't think you can be too paranoid about wonky logging or about unexpected user behavior.

Change 356169 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[mediawiki/extensions/CirrusSearch@master] [WIP] Explore similar widget for SpecialSearch

https://gerrit.wikimedia.org/r/356169

debt added a comment.May 30 2017, 8:12 PM

Latest screen shots:

Another note for event logging: we may want to include tracking of accept-language, especially for assessing the engagement with similar results from other languages.

debt updated the task description. (Show Details)Jun 8 2017, 6:05 PM
TJones updated the task description. (Show Details)Jun 8 2017, 6:06 PM
debt updated the task description. (Show Details)Jun 8 2017, 6:07 PM
debt updated the task description. (Show Details)
debt updated the task description. (Show Details)Jun 8 2017, 6:11 PM
debt updated the task description. (Show Details)Jun 8 2017, 6:14 PM
debt updated the task description. (Show Details)Jun 8 2017, 6:20 PM
debt updated the task description. (Show Details)Jun 8 2017, 6:27 PM
mpopov updated the task description. (Show Details)Jun 16 2017, 12:04 AM

Events

  • action = "hover-on" and "hover-off"
    • if the user moves mouse from one section to another (e.g. related pages to categories), that's two pairs of on-off hover events
    • in case of action == "hover-on", extraParams should have a stringified-JSON object of the following data:
      • hoverId: an ID generated randomly & saved upon hovering over a new result, used for pairing with "hover-off" events
      • section: "pages"|"categories"|"languages"
      • results: ## (0 if no results)
    • in case of action == "hover-off", extraParams should have a stringified-JSON object of the following data:
      • hoverId: the same ID used from the "hover-on" event, cleared upon hovering off until user hovers over another result
  • action = "esclick" for when the user clicks on one of the explore similar links
    • extraParams.hoverId = same ID used from "hover-on" event
    • extraParams.section = "pages"|"categories"|"languages" (used for consistency checking)
    • extraParams.result = # of the clicked result (e.g. 1 for 1st result) if in related pages or categories; name of clicked language otherwise

@chelsyx @Jdrewniak what do you think? Is there any interaction I forgot about?

Jan, I haven't changed https://meta.wikimedia.org/wiki/Schema:TestSearchSatisfaction2 yet because I wanted to get an OK from you and Chelsy about those events, since the schema would need to include "esclick" as a possible action.

Reminder that each link should have a wprov=??? appended to URL query (@EBernhardson should this follow the serp:N format as written in searchSatisfaction.js?)

Sampling

Based on ~2M full-text searchers (from T163273), our sampling will be 1 in 1000 to get into EL (so, roughly 2K searchers) and then half of those will go into the test and the other half will be used for metrics, yielding about 500 full-text searchers per group in test:

enwiki: {
    test: 1000,
    subTest: 2
}

Change 360851 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[mediawiki/extensions/WikimediaEvents@master] Adding event-logging for explore-similar on SRP

https://gerrit.wikimedia.org/r/360851

@mpopov those events sound good to me. In regards to the wprov we'll be using a modified version of the serp:N format (as discussed on IRC) with just _es appended to the end.

Change 360851 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Adding event-logging for explore-similar on SRP

https://gerrit.wikimedia.org/r/360851

Change 362265 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[mediawiki/extensions/WikimediaEvents@wmf/1.30.0-wmf.7] Adding event-logging for explore-similar on SRP

https://gerrit.wikimedia.org/r/362265

Change 362270 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[mediawiki/extensions/CirrusSearch@wmf/1.30.0-wmf.7] Explore similar widget for SpecialSearch

https://gerrit.wikimedia.org/r/362270

Change 356169 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Explore similar widget for SpecialSearch

https://gerrit.wikimedia.org/r/356169

Change 362270 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@wmf/1.30.0-wmf.7] Explore similar widget for SpecialSearch

https://gerrit.wikimedia.org/r/362270

Change 362265 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.30.0-wmf.7] Adding event-logging for explore-similar on SRP

https://gerrit.wikimedia.org/r/362265

Mentioned in SAL (#wikimedia-operations) [2017-06-29T23:26:46Z] <catrope@tin> Synchronized php-1.30.0-wmf.7/extensions/CirrusSearch/: "Explore similar" widget for CirrusSearch (T149809) (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2017-06-29T23:28:19Z] <catrope@tin> Synchronized php-1.30.0-wmf.7/extensions/WikimediaEvents/: Add event logging for explode-similar on SRP (T149809) (duration: 00m 42s)

Peter added a subscriber: Peter.Jul 3 2017, 5:39 PM

Change 364465 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[mediawiki/extensions/CirrusSearch@master] Adding full URLs to Explore Similar API calls

https://gerrit.wikimedia.org/r/364465

Change 364465 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Adding full URLs to Explore Similar API calls

https://gerrit.wikimedia.org/r/364465

Change 364834 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[mediawiki/extensions/CirrusSearch@wmf/1.30.0-wmf.9] Adding full URLs to Explore Similar API calls

https://gerrit.wikimedia.org/r/364834

Change 364834 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@wmf/1.30.0-wmf.9] Adding full URLs to Explore Similar API calls

https://gerrit.wikimedia.org/r/364834

Mentioned in SAL (#wikimedia-operations) [2017-07-12T23:28:44Z] <thcipriani@tin> Synchronized php-1.30.0-wmf.9/extensions/CirrusSearch/resources/ext.cirrus.explore-similar.js: SWAT: [[gerrit:364834|Adding full URLs to Explore Similar API calls]] T149809 T164856 (duration: 00m 47s)

debt closed this task as Resolved.Sep 22 2017, 2:30 PM
debt moved this task from UI to Done on the Discovery-Search (Current work) board.

We will not be going forward with this feature as both A/B tests did not show impactful usage of the related articles, categories or language links.

Here is the wrap-up T175647#3627249 from the second test that displayed language links to the same article written in a different language:

The analysis is done for the running of this test and unfortunately, we only had one clickthrough to an article written in a different language (which was displayed in the new language links) as the report documents. We will not be going forward with this feature.

However, if a user wants to have the language link script added to their logged-in account, please follow the instructions here: https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements/explore_similar#Explore_similar_-_for_languages_in_search_results.

The full explore similar script (displays related articles, categories and language links) can also be enabled for logged-in users, see this link for instructions: https://www.mediawiki.org/wiki/Cross-wiki_Search_Result_Improvements/self-guided_testing#Explore_similar