Page MenuHomePhabricator

Extend Teahouse retention analysis
Open, MediumPublic

Description

New questions:

  1. what are the specific 'triggers' that cause someone to visit the Teahouse? Can we infer from someone's editing patterns that they need help, and provide them an invite in a more timely fashion? Because previous research strongly suggests that Wikipedia loses a lot of potentially valuable new contributors within their first few edits/hours because they get confused and/or have a negative experience.
  2. who does the Teahouse NOT help? Can we identify new editors who experiencing challenges that the Teahouse, for whatever reason, does not seem to address for them? If we can identify valuable newcomers whose needs are probably not currently being met by the Teahouse (mobile-first users, to pick one example of many), then we can design and test interventions targeted at supporting those users.
  3. Does the Teahouse mitigate the effects of negative feedback on newcomers? The Teahouse was designed partially in response to work that Dr. Morgan and I did in 2011 while working as fellows for the Wikimedia Foundation. We showed that the negative feedback that newcomers receive is strongly correlated with decreased retention rates -- independent of their initial investment (# of edits in first session) and the quality of their work. It would be very useful to know if the Teahouse is especially effective in helping these users keep contributing.
  4. Are our technological interventions more effective on editors that are invited to the teahouse? I think that the reason we don't see significant gains in retention and productivity for VE is that the benefits of VE are too short-lived. Even if a newcomer has a better initial experience with editing, they're still likely to get negative feedback and leave. If the Teahouse reduces the social/motivation barriers to participation, then more immediate, technical interventions should have a larger apparent effect. See my sketch of entry barriers, VE and the Teahouse: https://www.mediawiki.org/w/index.php?title=File%3ATeahouse_research_showcase_October_2015.pdf&page=48

Event Timeline

Halfak created this task.Oct 22 2015, 10:23 PM
Halfak raised the priority of this task from to Needs Triage.
Halfak updated the task description. (Show Details)
Halfak added a project: Research.
Halfak moved this task to Staged (ready to execute) on the Research board.
Halfak added subscribers: Halfak, Capt_Swing.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 22 2015, 10:23 PM
DarTar assigned this task to Halfak.Oct 29 2015, 3:40 AM
DarTar triaged this task as Medium priority.
DarTar removed a subscriber: Halfak.
This comment was removed by chasemp.

@Capt_Swing started another experiment to gather more data. We'll need to wait ~1.5 months for that analysis.

Met with @Capt_Swing, I'm going to generate a dataset of invitation/warning messages posted on talk pages. We're going to use this to look at trends in messaging.

aripstra moved this task from In Progress to Paused on the Design-Research board.Dec 11 2015, 6:14 PM
DarTar moved this task from In Progress to Paused on the Research board.Jan 21 2016, 11:48 PM

Taking DR off this for now. It'll still live in the Teahouse project, if/when we have capacity to take this on, and decide it's a priority.