Page MenuHomePhabricator

Variant tests: "initiation part 2" test (C vs. D)
Open, Needs TriagePublic

Description

Goal

Increase the percentage of newcomers interacting with the and choosing articles from the suggested edits module, and ultimately increase the percentage of newcomers saving edits via the workflow. This distinction is important because it is theoretically possible for us to come up with designs that grab more people into the workflow, but don't give them context to know how to complete the workflow.

Background

When newcomer tasks was first introduced to the newcomer homepage, we conducted our first variant test: T238888: Variant tests: "initiation" test (A vs. B)

  • Variant A newcomers had to "initiate" the suggested edits module on their homepage by selecting a call to action in the Start module and proceeding through two onboarding overlays that introduced them to the content.
  • Variant B saw the suggested edits module right from their arrival ('pre-initiated') and didn't see the onboarding overlays.

Hypothesis

The first variant test taught us that a more prominent module can attract more interaction. But we also think it showed that onboarding (introducing the concept of "suggested edits") is an important part of converting newcomers who interact with suggested edits, to saving a suggested edit task. By combining the best of the two previous variants where users are both taken through onboarding (Variant A) and have prominent suggested edits content by default (Variant B), more newcomers will use suggested edits.

Takeaways from Var A vs B that inform this Var C vs D test:
1. More prominent SE module increases interaction.
This finding applied as best practice in variants C and D. Both focus on driving editing as the core goal of the newcomer experience. As part of this, the Start module will be removed to center onboarding and orientation on Suggested edits instead.
2. Onboarding is also important.
Despite variant B causing interactions to double, navigating only went up by 60%, and clicking only went up by 30%. It is inferred that whilst people seeing the pre-initated feature try it out, these ‘uninformed’ interactions cause more drop outs. This is where the onboarding info in variant A seems to play an important role in user engagement. This interpretation is supported by indicative results (i.e., not measured for statistical relevance) further down the funnel (edit attempts and completed edits are higher on var A).
3. Mobile SE preview design should be more actionable.
Mobile variant B performed slightly worse than variant A, meaning the SE preview card design isn’t encouraging discovery.

Proposal

Test two new variants, Variant C and Variant D, to see whether the emphasis on 'onboarding' content or prominence of the SE call to action leads to higher interaction and editing.
Mocks: https://wikimedia.invisionapp.com/overview/Variants-C-G-SE-top-of-funnel-tests-ck77r3arw00ea012a8bk1bh0v/screens?v=jXSLq0KIm0I3Xnr7YR5nPw%3D%3D
Redlined versions will be shared in the Growth Zeplin board (tagged with Variant-Pre-initiatedSE)

Note: the variants for mobile and desktop are not strictly comparable. They are similar in spirit, but can be essentially thought of as four different variants. This Phab task will refer to them as "Variant C-desktop", "Variant D-desktop", "Variant C-mobile", and "Variant D-mobile".

Specifications

Summary of new C & D flow changes

Related Objects

StatusSubtypeAssignedTask
OpenMMiller_WMF
OpenNone
OpenNone
OpenNone
OpenNone
Openkostajh
Openkostajh
Openkostajh
OpenNone
OpenNone
OpenTgr
OpenNone
OpenCatrope
Openkostajh
OpenTgr
OpenTgr
OpenCatrope
Openkostajh
OpenNone
OpenCatrope
DeclinedNone
OpenNone
OpenNone
Openkostajh
OpenNone
OpenNone
OpenNone

Event Timeline

RHo created this task.Feb 29 2020, 3:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 29 2020, 3:39 PM
RHo added a comment.Feb 29 2020, 3:40 PM

hi @MMiller_WMF please review the description and linked Invision mocks, then comment and update.

MMiller_WMF renamed this task from Newcomer tasks: Variant test - Pre-initiated SE module to Variant tests: "initiation part 2" test.Mar 1 2020, 9:21 PM
RHo updated the task description. (Show Details)Mar 4 2020, 9:56 AM
RHo updated the task description. (Show Details)Mar 4 2020, 4:55 PM
RHo updated the task description. (Show Details)Mar 4 2020, 6:48 PM
MMiller_WMF renamed this task from Variant tests: "initiation part 2" test to Variant tests: "initiation part 2" test (C vs. D).Mar 4 2020, 7:55 PM
MMiller_WMF updated the task description. (Show Details)
MMiller_WMF updated the task description. (Show Details)Mar 23 2020, 7:28 PM
RHo renamed this task from Variant tests: "initiation part 2" test (C vs. D) to [EPIC] Variant tests: "initiation part 2" test (C vs. D).Apr 7 2020, 8:53 PM
RHo added a project: Epic.
RHo updated the task description. (Show Details)
MMiller_WMF renamed this task from [EPIC] Variant tests: "initiation part 2" test (C vs. D) to Variant tests: "initiation part 2" test (C vs. D).Apr 8 2020, 6:38 PM
MMiller_WMF removed a project: Epic.
MMiller_WMF updated the task description. (Show Details)
MMiller_WMF updated the task description. (Show Details)Apr 16 2020, 5:56 AM
RHo updated the task description. (Show Details)Apr 29 2020, 5:54 PM
RHo updated the task description. (Show Details)May 1 2020, 10:58 PM