Page MenuHomePhabricator

Newcomer tasks: quick analysis of AND selection usage
Open, MediumPublic

Description

In T301825#7798417, we talked about how we'll be logging usage of the new AND filter for topics. We'll be deploying the feature to our pilot wikis, and after two weeks, we want to do a quick check of the usage.

Because this won't be a controlled experiment, we won't be able to tell whether the feature causes more people to make more edits. But we do want to check if it seems to be noticed, understood, and used by newcomers. We'll want to answer these questions:

  • What percent of users who encounter the topic selection screen (either while being onboarded or otherwise) change from "OR" to "AND" or vice versa?
  • What percent of users who exit the topic selection screen by selecting "Done" do so with the "AND" option selected?
  • Do users get stuck in a state that results in no suggested articles and then leave? We will define "leave" as not returning to the Homepage within the next 24 hours. Users who go back and change their settings are defined as "not left".

We want to break these numbers out by wiki and platform.

Event Timeline

MMiller_WMF renamed this task from Newcomer tasks: quick analysis of AND filter usage to Newcomer tasks: quick analysis of AND selection usage.Apr 4 2022, 8:08 PM

@MMiller_WMF - can we also determine if those who have switched to "AND" logic end up selecting articles to edit more so than those who select topics in the default OR mode?

nettrom_WMF added a project: Product-Analytics.

I suspect this will be assigned to me, so we might as well do that right away. Also adding Product Analytics so we can triage it and most likely drop it into the "Current quarter" column at our next board refinement meeting.

kzimmerman triaged this task as Medium priority.Apr 19 2022, 5:16 PM
kzimmerman moved this task from Triage to Current Quarter on the Product-Analytics board.

@nettrom_WMF wants to double-check that the fetch event happens when the user toggles "AND" and "OR", per T301825#7798417. i.e. checking whether the instrumentation implemented on T301825 is sufficient to do the analysis in this task.

@MMiller_WMF - can we also determine if those who have switched to "AND" logic end up selecting articles to edit more so than those who select topics in the default OR mode?

@RHo -- thanks for posting this idea. After talking with @nettrom_WMF, the way to really tell whether the "AND" filter is leading to improved outcomes would be to A/B test the presence of the option. We decided that we don't want to layer on an A/B test of the feature to the several other A/B tests we have running. But we can calculate the numbers as you described -- we would just have to caveat it with self-selection bias: people who flip the switch to "AND" may end up with higher editing rates just because they're the sort of people who interact with the software heavily.

@MMiller_WMF - can we also determine if those who have switched to "AND" logic end up selecting articles to edit more so than those who select topics in the default OR mode?

@RHo -- thanks for posting this idea. After talking with @nettrom_WMF, the way to really tell whether the "AND" filter is leading to improved outcomes would be to A/B test the presence of the option. We decided that we don't want to layer on an A/B test of the feature to the several other A/B tests we have running. But we can calculate the numbers as you described -- we would just have to caveat it with self-selection bias: people who flip the switch to "AND" may end up with higher editing rates just because they're the sort of people who interact with the software heavily.

OK, sounds good. I didn't think about it from that perspective but actually the other way, where the people who flip the switch to "AND" have lower editing rates because they more limited results and don't understand why.

@nettrom_WMF Do you have an ETA for being able to work on this task? I don't think we need a very heavy analysis here, more a quick check to ensure this feature isn't negatively impacting users in an unexpected way. Thanks!

Moving this off the PA Kanban board for now while we wait for it to unblock.

This is waiting for T312686 before data gathering can start. Moving it off the Growth sprint board until it gets unblocked.

This is waiting for T312686 before data gathering can start. Moving it off the Growth sprint board until it gets unblocked.

T312686 is now resolved and I think we could start gathering data from this Thursday on when the bugfix will reach production wikis.