Page MenuHomePhabricator

Add a link: background data analyses
Closed, ResolvedPublic

Description

As we think about the "add a link" structured task, we want to see what we can learn from newcomers' experiences with the existing non-structured "add a link" task type in the suggested edits module. Here are some potential questions we could answer. Their ranked importance is indicated alongside them (1 is highest).

2 -- Are newcomers attracted to the "add a link" task type?

Starting April 2 (per T248106), it's random whether a newcomer encountering suggested edits for the first time will see a "copyedit" task or an "add a link" task as their first one. Before April 2, all newcomers say a copyedit task first. And as they page through, the task types alternate. We want to know whether it seems like the copyedit tasks or the link tasks seem more attractive to newcomers, in terms of which gets more clicks. Two caveats are:

  • Korean Wikipedia does not have "add a link" tasks.
  • Czech Wikipedia has few "add a link" tasks, and therefore users in Variant A who select certain topics may end up with a copyedit task first if there are no link tasks available for their topics.

What this impacts: if we see low attraction to link tasks, we will consider how to accent their appeal in the module once there is structure behind the task.

3 -- How does the activation of the "add a link" task type compare to "copyedit"?

We want to know whether adding a link is an accessible task type that makes sense for newcomers to do. In other words, are users who select a link task first set up for success better than those who select a copyedit task first? Maybe we should look at: for newcomers who select a link task as their first task, how likely are they to edit that article, compared to newcomers who select a copyedit task as their first task.

What this impacts: if we see low activation to link tasks, we will be able to confirm that newcomers are confused by the task, given no structure. If the activation is high, we may second guess just how much structure is necessary for an improvement.

4 -- How does the retention of the "add a link" task type compare to "copyedit"?

We want to know whether adding a link is a particularly "sticky" task type that makes users want to do it again. In other words, are users who start out adding links more or less likely to keep adding links than the users who start out copyediting are to keep copyediting? Maybe we should look at: for users who select a link task as their first task and edit the article, how likely are they to do another link suggestion on a different article compared to how likely those who select a copyedit task first are to do another copyedit suggestion on a different article? What about just moving on to more suggested edits in general, regardless of type?

What this impacts: if we see high retention to this task, we'll have confirmation that newcomers can get into a groove with adding links, and we can design the feature to promote that behavior.

1 -- How do newcomers do the linking?

When users edit via a link task, how much linking do they tend to do? Do they add many links to the article? Or just one or two? Or perhaps they just end up doing copyediting, and not linking at all.

What this impacts: if we see newcomers adding many links throughout the article, then we might design the feature to encourage them to thoroughly link the article. If we see that they tend to do just one link and leave, then we might design the feature to encourage the newcomer to get a quick win and publish a simple, small edit. If their edits frequently add no links, then we might see that it is generally not understood how to add links, underscoring the need for guidance.

5 -- Bonus: how do clicks and edits compare across task types?

While we are doing the above analyses, perhaps it will be easy for us to look at all the five task types, to see how often each has been selected in the past, and how often each has gotten edits in the past.

Event Timeline

MMiller_WMF removed MMiller_WMF as the assignee of this task.May 14 2020, 8:49 PM
MMiller_WMF created this task.
RHo awarded a token.May 15 2020, 12:34 PM
MMiller_WMF updated the task description. (Show Details)May 15 2020, 5:06 PM
MMiller_WMF updated the task description. (Show Details)
LGoto assigned this task to nettrom_WMF.May 18 2020, 4:11 PM
LGoto moved this task from Triage to Needs Investigation on the Product-Analytics board.
MMiller_WMF updated the task description. (Show Details)May 21 2020, 2:38 AM
LGoto triaged this task as Medium priority.May 26 2020, 5:14 PM
LGoto moved this task from Needs Investigation to Current Quarter on the Product-Analytics board.

@MMiller_WMF : I unfortunately failed to write down where this fits in our list of priorities. Should I put this next up in my queue so I start it after the Newcomer Tasks analysis is ready for review?

@nettrom_WMF -- yes, that's right. This is the next priority after the newcomer tasks experiment.

nettrom_WMF moved this task from Current Quarter to Upcoming Quarter on the Product-Analytics board.

Moving this into the Upcoming Quarter board for the Product Analytics team, as given the Newcomer Tasks experiment analysis and Guidance instrumentation QA work, this will most likely not happen until Q1 of FY 20/21.

@MMiller_WMF : from what I know, this analysis should happen in Q1, hence I'm moving it into the Current Quarter board on the Product Analytics team to reflect that. Let me know if that assessment need adjustment.

@nettrom_WMF -- that's correct. This is what we want to work on after the newcomer tasks experiment analysis is complete.

@nettrom_WMF is going to be working on this in the next week.

Urbanecm edited subscribers, added: Urbanecm_WMF; removed: Urbanecm.Aug 26 2020, 2:10 PM
Tgr added a comment.Sep 8 2020, 2:47 AM

That is restricted to a small set of people. Can it be made public?

@Tgr -- I've changed it to be viewable by WMF users.

Isaac added a subscriber: Isaac.Sep 17 2020, 7:41 PM