Page MenuHomePhabricator
Create Task
Maniphest T305406

Newcomer tasks: quick analysis of AND selection usage
Closed, ResolvedPublic
Actions

Description

In T301825#7798417, we talked about how we'll be logging usage of the new AND filter for topics. We'll be deploying the feature to our pilot wikis, and after two weeks, we want to do a quick check of the usage.

Because this won't be a controlled experiment, we won't be able to tell whether the feature causes more people to make more edits. But we do want to check if it seems to be noticed, understood, and used by newcomers. We'll want to answer these questions:

  • What percent of users who encounter the topic selection screen (either while being onboarded or otherwise) change from "OR" to "AND" or vice versa?
  • What percent of users who exit the topic selection screen by selecting "Done" do so with the "AND" option selected?
  • Do users get stuck in a state that results in no suggested articles and then leave? We will define "leave" as not returning to the Homepage within the next 24 hours. Users who go back and change their settings are defined as "not left".

We want to break these numbers out by wiki and platform.

Related Objects
Search...

Event Timeline

MMiller_WMF created this task.Apr 4 2022, 7:53 PM
MMiller_WMF mentioned this in T305399: Newcomer tasks: deploy AND selection to pilot wikis.
MMiller_WMF mentioned this in T305097: Newcomer tasks: Copy and UI improvements the AND selection of interest topics.Apr 4 2022, 7:56 PM
MMiller_WMF renamed this task from Newcomer tasks: quick analysis of AND filter usage to Newcomer tasks: quick analysis of AND selection usage.Apr 4 2022, 8:08 PM
MMiller_WMF mentioned this in T305408: Newcomer tasks: deploy AND selection to all wikis.
RHo added a comment.Apr 5 2022, 10:12 AM

@MMiller_WMF - can we also determine if those who have switched to "AND" logic end up selecting articles to edit more so than those who select topics in the default OR mode?

nettrom_WMF claimed this task.Apr 18 2022, 3:28 PM
nettrom_WMF added a project: Product-Analytics.

I suspect this will be assigned to me, so we might as well do that right away. Also adding Product Analytics so we can triage it and most likely drop it into the "Current quarter" column at our next board refinement meeting.

kzimmerman triaged this task as Medium priority.Apr 19 2022, 5:16 PM
kzimmerman moved this task from Triage to Current Quarter on the Product-Analytics board.
MMiller_WMF added a comment.Apr 19 2022, 8:43 PM

@nettrom_WMF wants to double-check that the fetch event happens when the user toggles "AND" and "OR", per T301825#7798417. i.e. checking whether the instrumentation implemented on T301825 is sufficient to do the analysis in this task.

MMiller_WMF added a comment.Apr 19 2022, 8:48 PM
In T305406#7831359, @RHo wrote:

@MMiller_WMF - can we also determine if those who have switched to "AND" logic end up selecting articles to edit more so than those who select topics in the default OR mode?

@RHo -- thanks for posting this idea. After talking with @nettrom_WMF, the way to really tell whether the "AND" filter is leading to improved outcomes would be to A/B test the presence of the option. We decided that we don't want to layer on an A/B test of the feature to the several other A/B tests we have running. But we can calculate the numbers as you described -- we would just have to caveat it with self-selection bias: people who flip the switch to "AND" may end up with higher editing rates just because they're the sort of people who interact with the software heavily.

RHo added a comment.Apr 20 2022, 3:53 PM
In T305406#7865971, @MMiller_WMF wrote:
In T305406#7831359, @RHo wrote:

@MMiller_WMF - can we also determine if those who have switched to "AND" logic end up selecting articles to edit more so than those who select topics in the default OR mode?

@RHo -- thanks for posting this idea. After talking with @nettrom_WMF, the way to really tell whether the "AND" filter is leading to improved outcomes would be to A/B test the presence of the option. We decided that we don't want to layer on an A/B test of the feature to the several other A/B tests we have running. But we can calculate the numbers as you described -- we would just have to caveat it with self-selection bias: people who flip the switch to "AND" may end up with higher editing rates just because they're the sort of people who interact with the software heavily.

OK, sounds good. I didn't think about it from that perspective but actually the other way, where the people who flip the switch to "AND" have lower editing rates because they more limited results and don't understand why.

kostajh moved this task from Incoming to Ready for Development on the Growth-Team (Sprint 0 (Growth Team)) board.May 12 2022, 9:01 AM
kostajh changed the status of subtask T305408: Newcomer tasks: deploy AND selection to all wikis from Open to Stalled.May 12 2022, 9:04 AM
KStoller-WMF subscribed.Jun 10 2022, 4:12 PM

@nettrom_WMF Do you have an ETA for being able to work on this task? I don't think we need a very heavy analysis here, more a quick check to ensure this feature isn't negatively impacting users in an unexpected way. Thanks!

nettrom_WMF edited projects, added Product-Analytics (Kanban); removed Product-Analytics.Jun 21 2022, 5:14 PM
nettrom_WMF updated the task description. (Show Details)Jun 21 2022, 9:52 PM
Sgs changed the status of subtask T305408: Newcomer tasks: deploy AND selection to all wikis from Stalled to In Progress.Jun 28 2022, 11:36 AM
nettrom_WMF edited projects, added Product-Analytics; removed Product-Analytics (Kanban).Jul 5 2022, 5:08 PM

Moving this off the PA Kanban board for now while we wait for it to unblock.

nettrom_WMF edited projects, added Growth-Team; removed Growth-Team (Sprint 0 (Growth Team)).Jul 18 2022, 5:56 PM

This is waiting for T312686 before data gathering can start. Moving it off the Growth sprint board until it gets unblocked.

nettrom_WMF moved this task from Inbox to Upcoming Work on the Growth-Team board.Jul 18 2022, 5:56 PM
Sgs added a comment.EditedSep 12 2022, 5:41 PM
In T305406#8085513, @nettrom_WMF wrote:

This is waiting for T312686 before data gathering can start. Moving it off the Growth sprint board until it gets unblocked.

T312686 is now resolved and I think we could start gathering data from this Thursday on when the bugfix will reach production wikis.

nettrom_WMF moved this task from Current Quarter to Upcoming Quarter on the Product-Analytics board.Sep 13 2022, 5:13 PM
nettrom_WMF moved this task from Upcoming Quarter to Current Quarter on the Product-Analytics board.Oct 4 2022, 8:00 AM
nettrom_WMF edited projects, added Product-Analytics (Kanban), Growth-Team (Sprint 0 (Growth Team)); removed Growth-Team, Product-Analytics.Nov 6 2022, 10:59 PM
Urbanecm_WMF changed the status of subtask T305408: Newcomer tasks: deploy AND selection to all wikis from In Progress to Open.Nov 7 2022, 8:45 AM
nettrom_WMF moved this task from Ready for Development to In Progress on the Growth-Team (Sprint 0 (Growth Team)) board.Nov 7 2022, 4:44 PM
nettrom_WMF moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.
kostajh changed the status of subtask T305408: Newcomer tasks: deploy AND selection to all wikis from Open to Stalled.Nov 10 2022, 2:44 PM
Tgr changed the status of subtask T305408: Newcomer tasks: deploy AND selection to all wikis from Stalled to In Progress.Dec 14 2022, 2:18 AM
KStoller-WMF mentioned this in T325390: Suggested edits: Improve UX for "and" filtering leading to no suggestions found.Dec 16 2022, 7:43 PM
kostajh mentioned this in T326579: Suggested edits topic dialog: Disable save when in AND mode, and no results are found.Jan 9 2023, 5:40 PM
nettrom_WMF closed this task as Resolved.Feb 15 2023, 5:23 PM

We've done a first pass of this analysis by looking at users who exit the topic selection dialogue on wikis where this feature has been available. While that doesn't capture users who turn the feature on during the module initialization process, our findings suggested changes to the UX was needed (ref some of the related tasks from December and January) to improve the user experience. This conclusion wouldn't change if we extended the dataset, and so we decided to instead prioritize other work.

We used data from 2022-09-12 through 2022-10-31 from Arabic, Bangla, Czech, Persian, Portuguese, Spanish, and Turkish Wikipedia, and excluded known test accounts. Interactions with the topic selection dialogue are captured in the HomepageModule schema, which is a client-side schema that can be blocked by ad-blockers and other software. That is a known limitation of this dataset.

Overall, we found that 11.8% of desktop users and 6.5% of mobile users tried out this feature. Usage does vary by wiki, but the larger proportion of usage on desktop is consistent.

We also found that those who exit the topic selection dialogue with AND enabled and in a state that results in zero available tasks are unlikely to fix the problem by reopening the dialogue, and they are also unlikely to return to the Homepage within the next 24 hours. This finding suggested that we change the dialogue so that exiting in a state with zero available tasks would not occur, and that work was done in T325390.

KStoller-WMF mentioned this in T335205: Rollback AND topic selection on Pilot wikis.Apr 21 2023, 7:32 PM
KStoller-WMF closed subtask T305408: Newcomer tasks: deploy AND selection to all wikis as Declined.
Urbanecm_WMF removed a subscriber: MShilova_WMF.Jul 31 2023, 6:04 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL