Page MenuHomePhabricator

[SPIKE] Investigate why we are seeing people bypassing the Reference Check decline survey
Closed, ResolvedPublic

Description

In T405421#11441696, we learned that in ~10% of edits where Reference Check was shown, as part of the A/ experiment ongoing at en.wiki (T400101), people are dismissing Reference Check without providing any reason.

This task involves the work of understanding how/why this might be so considering we've designed the decline survey to require people to complete it.

Open question(s)

  • 1. What could explain why 11% of the Reference Check "dismissals" we are seeing as part of T400101 do not include any associated decline reason?

Done

  • Answers to all open question are documented

i. "We expected every dismissal to include one of four required reasons (Other, Irrelevant, Uncertain, or Common Knowledge), since users must select a reason when dismissing Reference Check. In practice, we see cases where Reference Check was shown and the user dismissed; 11% of dismissals have no reason provided."

Event Timeline

The sequence of events is:

  • editCheck-addReference : action-reject - the user chose "no"
  • editCheck-addReference : edit-check-feedback-shown - the feedback form was shown
  • editCheck-addReference : edit-check-feedback-reason-[reason] - the user chose reason in the form

Notably, there's a "back" button on the feedback form. The user can click it rather than choosing an option -- it won't dismiss the check, but it'll let you go back and either pick another option or go even further back and edit the content through normal means. However, it's not instrumented when people choose it.

So: this 11% is either people who have gone back to choose another option, or who have given up there-and-then and closed the window.

So: this 11% is either people who have gone back to choose another option, or who have given up there-and-then and closed the window.

To be doubly sure, I'm following...

When you say, "people who have gone back to choose another option..." are you referring to people who follow the following sequence?

  1. See Reference Check
  2. Decline to add a reference
  3. See the feedback form
  4. Tap Back from within the feedback form
  5. Elect to add a reference

Yes, though as well as "5. add a reference" they could also have either removed the triggering content from their edit, or abandoned their edit entirely.

(Analysis of the edit session would let you broadly determine which of those was the case, though it'd be pretty in-the-weeds, and you may not be able to tell the difference between removing the content and adding the reference manually if the user was being served multiple reference checks.)

ppelberg claimed this task.

@DLynch: mmm, got it. Okay. I appreciate you walking out what would be involved with investigating this further. For now, I think we can consider this resolved.