Page MenuHomePhabricator

[A/B Test] Add thumbnail icons to the search results
Open, NormalPublic

Description

Based on the finalized design from T146663 and T139310, this A/B test will have a test group that will be shown thumbnails (free use images only) for the search query results and updated metadata for each search result (that have images to display). A control group will see the currently existing search results page.

This test is expected to last a week - the Wikipedia where this test will be run on is yet to be determined, but will be based on community input.

Take into account this community feedback on using Wiktionary results.

Note: we'll need to see if the images increase the clickthroughs on search results for the listing and how long the page took to load.

Note: The images used in this test will be displayed using the PageImages API extension, similar to what is being used on the Wikipedia portal page in the search box type ahead feature.

Test group users will see for all search results returned:

  • thumbnail (as available) image for the search results

Test group users will see for all search results returned:

  • normal search results page

Bucket testing logic generally is as follows:

  • 1 in 200 users are included in EventLogging
  • Of those 1 in 200 users, 1 in 10 are included in the test
  • Of those 1 in 10 users
    • 50% go in a test group, labeled "serp_add_thumbnails"
    • the other 50% of users will go in a control group, labeled "serp_no_thumbnails"
  • The remaining chunk of the original bucketed 200 users will get a NULL (the string null, or the MySQL null, we can detect either).

Eventlogging needs to capture:

  • if the user clicked on any of the top 3 search results vs the rest
  • what position the result was in that the user clicked on (1st, 2nd or 3rd search result)
  • how long it took the images to load on the page
  • how many images were displayed as compared to the amount of the search results returned

First draft:

Second draft:

Event Timeline

debt created this task.Nov 2 2016, 3:33 PM
debt edited projects, added Discovery-Search (Current work); removed Discovery-Search.
debt moved this task from in progress to UI on the Discovery-Search (Current work) board.
debt added a comment.Nov 15 2016, 1:22 AM

This should be blocked on fixing T95026: PageImages should be able to pick Wikidata's P18 as chosen image.

We'll use the standardized approach - using the PageImages API - that we're using in other places. It would be great if T95026 would be fully fixed, but I don't think it's a blocker to this A/B test.

debt updated the task description. (Show Details)Jun 2 2017, 10:45 PM
debt updated the task description. (Show Details)Jul 20 2017, 7:26 PM
debt assigned this task to Jdrewniak.Jul 25 2017, 3:48 PM
debt renamed this task from [A/B Test] Add thumbnail icons to top 3 search results to [A/B Test] Add thumbnail icons to the search results.Jul 25 2017, 3:53 PM
debt updated the task description. (Show Details)
debt updated the task description. (Show Details)Jul 25 2017, 3:56 PM
debt updated the task description. (Show Details)Jul 27 2017, 2:57 PM

I'm now wondering: will the images be clickable? (In theory they should be, to give attribution.) Images are click-traps, so I expect it will be useful to know whether they "steal" some clicks from actual search results.

In short: please count clicks on images and clicks on pages, but separately.

I also wonder how to measure and assess whether the images make people click different search results: the screenshot shows the first three results as having an image, but I guess it could also happen that the first result has no image and the following three have. I'd expect users to become more likely to click the non-first results that they would otherwise. But then this doesn't tell us whether the image-endowed result was actually better or worse for them.

Maybe recording the number of images displayed and the position of the clicked results (which you already do or plan to do, AFAICT) will be enough for later analysis if needed.

@Nemo_bis In this context, thumbnails serve just as a visual cue for the search result, and don't have any intrinsic value on their own. This is why on mobile search (typeahead) and on other search engines, clicking on the image acts just like clicking on the text. The thumbnail + search result are perceived as one unit, so it would be confusing if the text led you to one page but the image to another.

In the code I've written so far for this test, I've made the image not clickable (mostly for simplicity), but if they were clickable, I think it would make the most sense to make them lead to the search result page as well. Given that the page-images are probably the most prominent image on an article page, I'd leave it to users to click through to the image from the there if they are interested in it.

recording the number of images displayed and the position of the clicked results

Yes, this is what we plan to do. we'll record the number of images displayed, and their position, as well position of the clicked result (which we do already).

A demo of this feature is available for testing by adding the following line to a users common.js file on english wikipedia:
importScript( 'User:JDrewniak_(WMF)/searchThumbs.js' );

Cool demo Jan. Now I want the image to link to the article. :)

@mpopov regarding the event logging for this test, I'd like to go over some details of how information about these thumbnails can be collected.
I have an idea that I think integrates nicely with the existing SearchSatisfaction schema.

Currently, when a user is selected for a test group, a searchResultPage event is logged when the search page is loaded. This event already contains information about the results presented within the extraParams property (number & order of sister search results, pagination offset, etc). For this test, I propose we add a value to extraParams for the searchResultPage event named something like thumbnailPositions that would contain an array of the positions of the thumbnails, so [3, 5, 10] would mean thumbnails were presented for the 3rd 5th 10th search results. The length of the array would give us the total number of thumbnails presents, and when a user clicks through to a result, since we already log which position was clicked, we can just correlate whether that position had a thumbnail or not.

Does that seem like a sound idea?

I'm now wondering: will the images be clickable? (In theory they should be, to give attribution.)

Not necessarily.

If a free image is used in a thumbnail on a search result, then the image licensing for that image is two clicks away from where the image is displayed, i.e. click on the result to go to article, then click on the image in the article. This is well within the typically accepted bounds of sufficient attribution for most (if not all?) free licences. For fair use images, attribution technically isn't required at all, so two clicks would also be acceptable. There's not a big difference between two clicks (as in the search results) and one click (as in articles), especially when it's clear that the first click will take you to the article in question which contains the image.

(I am not a lawyer, I'm just someone who's dealt with this exact problem before, i.e. what is sufficient attribution for images and text in terms of how many clicks away it is from another page.)

@mpopov regarding the event logging for this test, I'd like to go over some details of how information about these thumbnails can be collected.
I have an idea that I think integrates nicely with the existing SearchSatisfaction schema.
Currently, when a user is selected for a test group, a searchResultPage event is logged when the search page is loaded. This event already contains information about the results presented within the extraParams property (number & order of sister search results, pagination offset, etc). For this test, I propose we add a value to extraParams for the searchResultPage event named something like thumbnailPositions that would contain an array of the positions of the thumbnails, so [3, 5, 10] would mean thumbnails were presented for the 3rd 5th 10th search results. The length of the array would give us the total number of thumbnails presents, and when a user clicks through to a result, since we already log which position was clicked, we can just correlate whether that position had a thumbnail or not.
Does that seem like a sound idea?

Sounds good! And if you decide to make the images clickable after all, you can likewise include a boolean thumbnail flag in click events' extraParams :)

Change 373049 had a related patch set uploaded (by Jdrewniak; owner: Jdrewniak):
[mediawiki/extensions/WikimediaEvents@master] [WIP] Search thumbnails AB test

https://gerrit.wikimedia.org/r/373049

debt updated the task description. (Show Details)Aug 22 2017, 3:12 PM

Moving this to the backlog for now, as other priorities have surfaced.

If a user wants to add this into their logged-in account, simply add this script to their common.js file:

importScript( 'User:JDrewniak_(WMF)/searchThumbs.js' );

Change 373049 abandoned by Jdrewniak:
Search thumbnails AB test

Reason:
we won't be going forward with this AB test.

https://gerrit.wikimedia.org/r/373049