In our analysis of the funnel for "add a link", we saw that users who completed the onboarding dialogs were substantially more likely to complete the rest of the workflow and wind up making a link edit. This effect may be due to either of two phenomena or a combination of them:
- The onboarding content helps users understand how the task works, leading them to success.
- Users who are attentive and conscientious enough to complete onboarding are the same kind of users that are attentive and conscientious enough to complete the task.
We propose an experiment in which a portion of "add a link" users are not given onboarding at all, and the other portion are not given the option to skip onboarding. From this, we would want to discover:
- Which group has better completion rates for each stage of the funnel ("link decision", "task review", "task completion").
- Which group has higher edit volume across all sessions?
- Which group has higher revert rate on their first session? Across all sessions?
- Which group is more likely to do a second task after completing their first?
- Which group has a higher rate of opening the help content during the task?
- For the group that cannot skip onboarding, how does the bounce rate from onboarding compare to the previous cohort that could skip?
For the group that does not have the option to skip onboarding, we would also want to make this design change: the "don't show again" checkbox would be on the third onboarding screen, not the first. This task to be created as a subtask.
We'll also need to consider how the escape key behaves for each variant, and whether it allows the onboarding dialog to be dismissed.
