Page MenuHomePhabricator

[Spike 1 day] Identify delivery mechanism of experiment pt 1
Closed, ResolvedPublic2 Estimated Story PointsDesign

Description

Background

  • We would like to run an experiment on the UI of the empty state of search. This ticket will cover investigating different options for the set up for this experiment and making a recommendation for the best option

User story

  • As a developer, I would like to know what the best way to deliver an experimental version of the empty search state is so that I can understand whether the empty state of search is an effective placement for recommendations

Requirements

NOTE: to be used in pt2. Leaving here for reference

Metrics requirements

The experiment designed should allow us to track the following (or as much as the following as possible)
Quantitative metrics - high priority

  • Search sessions initiated
  • Impressions of search suggestion
  • Clicks on suggestions
  • Text types into search - i.e. we would want some way to know whether a session was completed using a suggestion, completed using search (they began typing and selected a result), or abandoned

Qualitative metrics - nice to have

  • Overall interest in search suggestions
  • Overall satisfaction with UI of feature
  • Ease of use of functionality itself

Other constraints

  • The experiment should take no more than 2 weeks to build and set up
  • The experiment cannot run for more than 2 weeks
  • Nice to have - a group large enough to test on as to render the results statistically significant
  • Nice to have - the ability to compare the results to similar results from features such as related pages
  • Nice to have - the ability to measure external referrals for the pages of the test

Acceptance criteria

  • Create a list of previously completed experiments/ways this or other teams has presented new features to users/(adding new ideas of potential ways we could do this is also acceptable)
  • Document this list

Communication criteria - does this need an announcement or discussion?

  • Define communication criteria for experiment

Rollback plan

  • What is the rollback plan in production for this task if something goes wrong?

This task was created by Version 1.2.0 of the Web team task template using phabulous

Event Timeline

ovasileva triaged this task as High priority.

@ovasileva to split ticket AC1 and AC2 will be ticket 1, AC3 will be ticket 2

ovasileva renamed this task from [Spike] Identify delivery mechanism of experiment to [Spike] Identify delivery mechanism of experiment pt 1.Jul 29 2024, 5:45 PM
ovasileva updated the task description. (Show Details)
ovasileva renamed this task from [Spike] Identify delivery mechanism of experiment pt 1 to [Spike 1 day] Identify delivery mechanism of experiment pt 1.Jul 29 2024, 5:51 PM
ovasileva set the point value for this task to 2.

@dchen and I thought up a potential low-code solution to this earlier today. I listed it out in T371231:

I just had a great conversation with @dchen about this. Here is the recommendation to get directional data that answers the question: "Is the empty state of Search an effective location to show recommendations?"

We handpick a series of articles and launch a survey of some kind on those articles. We could also do all articles, or a subset based on some logic. Maybe trending articles?
In the survey, we show mockups of 3-5 different places where we could show recommendations on those pages. The mockups don't even have to show the content of the recommendations themselves. We could redact them somehow or use placeholder text.
We ask participants either to pick their top preference, or their top 2, or however many. We could also ask them to rank the different locations:

  1. Empty state of search
  2. Footer like in Minerva
  3. Appended to an infobox
  4. Sidebar like where the appearance menu is
  5. Maybe in the hatnote location above the introduction paragraph
  6. And we can think up a few more.

Nice to have would be for the order of the locations to be randomized for each participant to avoid cognitive bias.

This would give us a sense of the stated reader preference for different locations. It won't give us what they would actually do in a real reading session, or whether they would click on recommendations at all. But if we are going to show recommendations somewhere, it could give us the location that would have the highest yield.

Can we view quick survey results without logging in and/or agreeing to T&Cs of some kind?

@JScherer-WMF from the documentation it looks like QuickSurvey results are sent to eventLogging analytics platform, so accessing the results does require special permissions (generally limited to WMF staff) and might require help from analytics, but the aggregate results are usually made public in a report of some kind.

Jdrewniak subscribed.

Given the second part of this spike is underway using the survey provided above, I'm moving this into sign off.

Jdlrobson subscribed.

Jan to setup a meeting to make a decision.

Discussed and made decision on next steps. See task graph for details. Resolving this one!