Page MenuHomePhabricator

Personalized first day: instrument survey form
Closed, ResolvedPublic

Description

We will want to have insight into new users' interactions with the form we're designing. Beyond the actual answers that users give via the survey form, here are the additional things we will want to know:

  1. How often do users skip the form without filling any of it out?
  2. How often do users fill out some of the questions before dismissing/skipping the form? Which questions?
  3. How often do users click on other links that are contained in the form?
  4. How do activation rates (or other behavior) change between users who do and do not receive the form, according to the experiments laid out in T206380?

For #3, we be able to get that from the "Understanding first day" instrumentation.

#1 seems to be the open question.

We will also want to connect this with the data we are gathering via "Understanding first day" so that we can see the impact that this feature has on the broader experience for the new editor.

The first question is for @SBisson -- will we be capturing these things via the current plan? Will any additional instrumentation be necessary?

Details

Related Gerrit Patches:
mediawiki/extensions/GrowthExperiments : masterAdd parameter to distinguish source of links
mediawiki/extensions/GrowthExperiments : wmf/1.33.0-wmf.4Add parameter to distinguish source of links

Event Timeline

nettrom_WMF moved this task from Triage to Next Up on the Product-Analytics board.Oct 11 2018, 8:10 PM
MMiller_WMF updated the task description. (Show Details)
MMiller_WMF moved this task from FY 2019-20 to Upcoming Work on the Growth-Team board.

If @SBisson answers the questions on here in the affirmative, we might not need to do any more work with this task. @nettrom_WMF please chime in if you see things differently.

  1. How often do users skip the form without filling any of it out?

We can capture the use of the skip button but there are other ways to skip: the user may close the page, click on a link in the left nav, enter facebook dot com in the address bar, etc. If we measure conversion with the submitted responses, everybody else in the target group can be assumed to have skipped after a reasonable time (24h?)

  1. How often do users fill out some of the questions before dismissing/skipping the form? Which questions?

This one is tricky. I was assuming that we would not capture any of the data if the user decides to skip. Is it ok to capture that they have provided some data but not the data itself?

We can also add how long they've been on the form before submitting or skipping.

  1. How often do users click on other links that are contained in the form?

Yes, this is captured by the PageViews instrumentation (understanding first day).

RHo added a comment.Oct 19 2018, 2:10 PM
  1. How often do users skip the form without filling any of it out?

We can capture the use of the skip button but there are other ways to skip: the user may close the page, click on a link in the left nav, enter facebook dot com in the address bar, etc. If we measure conversion with the submitted responses, everybody else in the target group can be assumed to have skipped after a reasonable time (24h?)

I'm not so worried about users that enter in another URL to skip, but is it possible to also capture when users click on other links on the page that are *outside* of the form but still within the page, eg. the user searches for an article instead or clicks on "Random article" in the side nav? Knowing that would help determine how much the amount of Special page chrome deters completion (assuming we start off with the var A design in T206372)...

The PageViews schema will capture any navigation using the links on the standard chrome.

Running an experiment to figure out how much the chrome affects completion rate is interesting but I don't know if it's realistic this quarter.

Regarding question 1, applying a heuristic that defines that the user abandoned the form seems applicable, within a "reasonable time" as @SBisson suggests. Given that they're shown the form immediately after creation, I would consider applying the one hour rule for session abandonment here (e.g. an edit session is abandoned if more than one hour elapses before the next edit), there's plenty of precedent in the literature for that window.

MMiller_WMF updated the task description. (Show Details)Oct 20 2018, 12:01 AM

I hadn't thought of capturing when users abandon the form by closing the tab or going to a different URL. @SBisson are you proposing a way to do that without an EventLogging schema?

I hadn't thought of capturing when users abandon the form by closing the tab or going to a different URL. @SBisson are you proposing a way to do that without an EventLogging schema?

@nettrom_WMF and I are saying that after an hour, we can assume a user has abandoned.

When we look at those users and try to understand what happened, the PageViews data can tell us a lot. If the survey is the last page viewed by the user, they've probably closed the page or navigated to an external site explicitly. If there's other wikipedia pages after the survey, they've probably navigated away using some of the other page links.

MMiller_WMF closed this task as Resolved.Oct 30 2018, 12:25 AM

Okay, I think we are going to capture all the data we want from the survey form without any EventLogging. I am resolving this task.

@SBisson @kostajh @RHo -- in a conversation today, @DannyH pointed out that the same "Tutorial" and "Help desk" links are in two places in the survey. They're on the righthand side when taking the survey, and they're on the "Thank you" page. We also noticed that both pages are called "Special:WelcomeSurvey". Since our strategy for finding out whether people clicked those links is through the EditorJourney schema, do you think we'll be able to tell whether they clicked them from the survey page or from the "Thank you" page?

RHo added a comment.Nov 10 2018, 8:42 AM

Oh good catch. @SBisson can each link when clicked from the thank you page have a parameter appended to indicate this? Something like ?source=thankspg

Change 473237 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[mediawiki/extensions/GrowthExperiments@master] Add parameter to distinguish source of links

https://gerrit.wikimedia.org/r/473237

SBisson reopened this task as Open.Nov 13 2018, 4:07 PM
SBisson edited projects, added Growth-Team (Current Sprint); removed Growth-Team.
SBisson moved this task from Incoming to Code Review on the Growth-Team (Current Sprint) board.

I've submitted a patch to add ?source=survey for the sidebar and ?source=confirmation for the confirmation page.

I'll make sure to add the same parameter with different values for variation C.

Change 473237 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Add parameter to distinguish source of links

https://gerrit.wikimedia.org/r/473237

Change 473845 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[mediawiki/extensions/GrowthExperiments@wmf/1.33.0-wmf.4] Add parameter to distinguish source of links

https://gerrit.wikimedia.org/r/473845

Change 473845 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.33.0-wmf.4] Add parameter to distinguish source of links

https://gerrit.wikimedia.org/r/473845

Etonkovidova closed this task as Resolved.Nov 20 2018, 12:46 AM