Page MenuHomePhabricator

Initial analysis of the introduction of the talk page tab for mobile anons
Open, MediumPublic

Description

Background

In T293946: Enable talk for mobile users on enwiki we enabled the talk page tabs on the mobile website for anonymous users. We would like to analyze the effects of these changes and answer a number of research questions. This task, however, is intended to provide only a snapshot of the data gathered within the first two weeks of deployment. In-depth analysis will be tracked elsewhere (tasks to be created)

Acceptance criteria

Provide a brief overview of the following:

Event Timeline

ovasileva triaged this task as Medium priority.Nov 18 2021, 10:50 AM

@jwang - this is the first of two analysis tasks. This one is just for a quick overview of the main metrics from what we can see in superset/turnillo. Full analysis for the reading and editing side of the impacts will be tracked in other tickets

Revert rates on talk pages for mobile before and after the change

After enable talk page to mobile anonymous user on Nov 15, 2021 (46th week), the revert rate increased from 10.46% in the 45th week to 20.17% in 46th week and stayed at the higher level for the following weeks. It's because the revert rate is higher on anonymous users than logged-in users. The growth of edits from mobile anonymous user lifted the overall revert rate on talk page.

image.png (1×1 px, 297 KB)

Note: we defined reverts as the number of edits which were reverted within 48 hours after editing

Bounce rates on talk pages for mobile

Bounce rate is defined as number of sessions without any clicks out of total sessions initiated on talk pages. We deployed instrumentation after talk page was enabled. Therefore, we do not have data before the change.

Below graph shows the bounce rate on talk page for all users after talk page was enable on Nov 15, 2021. By the end of Dec 2021, the bounce rate varied between 54.35% and 60.68%, with an average of 57.91%. The average bounce rate of logged-in users is 37.00% between Nov 15, 2021 and Dec 31, 2021. If we assume the bounce rate of logged-in users keep same pre and post the change, we estimate that bounce rate increased 20.91% due to enabling talk page for mobile web anonymous users.

No obvious improvement on bounce rate has been observed in the following 6 weeks.

image.png (1×1 px, 270 KB)

Pageviews for talk pages on mobile before and after the change
  • The pageviews of talk pages from mobile web logged-out users increased significantly since Nov 15, 2021
  • The pageviews of talk pages from mobile web logged-out users has started to grow before the official enable date. It started to grow on Nov 8, 2021, and spiked between Nov 09, 2021 and Nov 11,2021 for the first time. We are not aware of the reason which caused the spike before Nov 15, 2021.
  • The pageviews of talk pages from mobile web logged-out users is not 0 before the official enable date. It might be due to mis-categorization of access method.
  • Based on the above observation, we select 2 weeks between 2021-10-24 and 2021-11-06 as the pre-deployment stage. We selected 2 weeks between 2021-11-28 and 2021-12-11 as the post-deployment stage. We estimated that the daily pageviews grew 195k due to enabling talk page to mobile web logged-out users.

image.png (1×1 px, 189 KB)