Page MenuHomePhabricator

Variant tests: monitoring dashboard for C vs. D
Closed, DeclinedPublic

Description

The Growth team is going to deploy Variants C and D, and then plans to do a formal analysis of their impact five weeks later via T264831: Variant tests: C vs. D analysis. In the meantime, @nettrom_WMF had the idea to make a monitoring dashboard via Superset/Presto so that we can notice drastic changes in user behavior in advance of that test. Such a dashboard would look at the same metrics as the experiment analysis would:

  • Visits (mobile only): what percent of newcomers visit the full suggested edits module on mobile? This only applies on mobile because mobile users must tap the module preview or go through onboarding before getting to the full module.
  • Interaction: what percent of newcomers interact with the suggested edits module? Note that we want to only count interactions with the fully-initiated suggested edits module. Interacting with topic selection and difficulty selection during the onboarding for Variant D does not count.
  • Navigation: what percent of newcomers navigate to multiple tasks in the module?
  • Task selection: what percent of newcomers click on a task to do?
  • Edit success: what percent of newcomers save an edit?

Event Timeline

Now that our analysis of these variants is complete, we no longer need a dashboard.