Page MenuHomePhabricator
Create Task
Maniphest T210868

Personalized first day: experiments (Variation C)
Closed, ResolvedPublic
Actions

Description

Once Variation C is ready, we should plan for experiments we will do with that variation.

This will result in modifications to the written experiment plan to indicate what share of users should receive Variation A, Variation C, and no survey. That can be up to @nettrom_WMF and @SBisson.

Related Objects
Search...

Event Timeline

MMiller_WMF created this task.Nov 30 2018, 6:30 PM
MMiller_WMF mentioned this in T206380: Personalized first day: experiments (Variation A).
MMiller_WMF edited projects, added Growth-Team (Sprint 0 (Growth Team)); removed Growth-Team.
MMiller_WMF moved this task from Incoming to Ready for Development on the Growth-Team (Sprint 0 (Growth Team)) board.
Aklapper removed a subscriber: Growth-Team.Dec 7 2018, 12:53 PM
nettrom_WMF moved this task from Ready for Development to In Progress on the Growth-Team (Sprint 0 (Growth Team)) board.Jan 2 2019, 6:40 PM
nettrom_WMF mentioned this in T212799: Personalized first day: determine the survey's effect on editor activation rate.Jan 2 2019, 6:56 PM
nettrom_WMF added a comment.Jan 2 2019, 11:15 PM

I've discussed this with both the Product Analytics team as well as @MMiller_WMF and we've decided that the experiment with Variation C will depend on the outcome of our A/B test of Variation A, specifically how the survey affects editor activation rate (which is measured in T212799).

  • If our analysis find that there is no significant difference in activation rate between Variation A and the control group, then we run a one-month experiment comparing Variation A against Variation C. If we find after that month that Variant A is worse than Variant C, then we run a subsequent test of Variant C against a control group.
  • If our analysis finds that Variant A resulted in a significant reduction in activation rate, then we run a one-month A/B test of Variation C against a control group.
MMiller_WMF mentioned this in T207717: Personalized first day: build Variation C.Feb 5 2019, 7:56 PM
nettrom_WMF added a project: Product-Analytics.Feb 27 2019, 5:12 PM
nettrom_WMF moved this task from Triage to Doing on the Product-Analytics board.
MMiller_WMF added a comment.Mar 28 2019, 11:57 PM

Leaving this in progress because the next step is to get results from this experiment on mediawiki.org. This is not urgent.

kzimmerman triaged this task as Low priority.Sep 11 2019, 9:42 PM
• mmodell edited projects, added Product-Analytics (Kanban); removed Product-Analytics.Oct 16 2019, 6:32 PM
• mmodell moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Oct 16 2019, 6:32 PM
kzimmerman moved this task from Kanban to Current Quarter on the Product-Analytics board.Feb 12 2020, 10:55 PM
kzimmerman edited projects, added Product-Analytics; removed Product-Analytics (Kanban).
kzimmerman closed this task as Resolved.Mar 18 2020, 7:05 PM
kzimmerman added a subtask: T248015: Welcome Survey: Document variant test findings.
kzimmerman subscribed.

Remaining documentation moved to a separate task, closing the analysis as resolved.

Urbanecm edited subscribers, added: Urbanecm_WMF; removed: Urbanecm.Aug 26 2020, 3:25 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits