Initial analysis of the introduction of the talk page tab for mobile anons
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ovasileva
	Nov 18 2021, 10:49 AM

Description

Background

In T293946: Enable talk for mobile users on enwiki we enabled the talk page tabs on the mobile website for anonymous users. We would like to analyze the effects of these changes and answer a number of research questions. This task, however, is intended to provide only a snapshot of the data gathered within the first two weeks of deployment. In-depth analysis will be tracked elsewhere (tasks to be created)

Acceptance criteria

Provide a brief overview of the following:

Revert rates on talk pages for mobile before and after the change made in T293946: Enable talk for mobile users on enwiki
Bounce rates on talk pages for mobile as defined in T294738: Define and instrument bounce rate on talk pages
Pageviews for talk pages on mobile before and after the change made in T293946: Enable talk for mobile users on enwiki

Analysis code: https://github.com/jenniferwang-wmf/WEB_talk_pages_to_mobile_anons/blob/master/Analysis.ipynb

Related Objects
Search...

Status	Assigned	Task
Open	None	T158181 Aim for workflow equivalence for MediaWiki on desktop and mobile web
Open	None	T54165 Links to talk pages in mobile view for all anonymous users
Resolved	ovasileva	T293946 Enable talk for mobile users on enwiki
Open	None	T276497 Scale DiscussionTools to all projects
Open	None	T251207 [Epic] Scale DiscussionTools to all Wikipedias
Open	None	T313976 [Release Ticket] Offer complete set of DiscussionTools features to to all wikis on mobile
Open	ovasileva	T294503 [EPIC] Measuring the impact of exposing talk pages to mobile web anons
Resolved	jwang	T295961 Initial analysis of the introduction of the talk page tab for mobile anons

Event Timeline

ovasileva created this task.Nov 18 2021, 10:49 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 18 2021, 10:49 AM

@jwang - this is the first of two analysis tasks. This one is just for a quick overview of the main metrics from what we can see in superset/turnillo. Full analysis for the reading and editing side of the impacts will be tracked in other tickets

ovasileva moved this task from Incoming to Analyst Consultation on the Web-Team-Backlog board.Dec 3 2021, 3:09 PM

ppelberg subscribed.Jan 12 2022, 9:26 PM

Revert rates on talk pages for mobile before and after the change

After enable talk page to mobile anonymous user on Nov 15, 2021 (46th week), the revert rate increased from 10.46% in the 45th week to 20.17% in 46th week and stayed at the higher level for the following weeks. It's because the revert rate is higher on anonymous users than logged-in users. The growth of edits from mobile anonymous user lifted the overall revert rate on talk page.

Note: we defined reverts as the number of edits which were reverted within 48 hours after editing

Bounce rates on talk pages for mobile

Bounce rate is defined as number of sessions without any clicks out of total sessions initiated on talk pages. We deployed instrumentation after talk page was enabled. Therefore, we do not have data before the change.

Below graph shows the bounce rate on talk page for all users after talk page was enable on Nov 15, 2021. By the end of Dec 2021, the bounce rate varied between 54.35% and 60.68%, with an average of 57.91%. The average bounce rate of logged-in users is 37.00% between Nov 15, 2021 and Dec 31, 2021. If we assume the bounce rate of logged-in users keep same pre and post the change, we estimate that bounce rate increased 20.91% due to enabling talk page for mobile web anonymous users.

No obvious improvement on bounce rate has been observed in the following 6 weeks.

Pageviews for talk pages on mobile before and after the change

The pageviews of talk pages from mobile web logged-out users increased significantly since Nov 15, 2021
The pageviews of talk pages from mobile web logged-out users has started to grow before the official enable date. It started to grow on Nov 8, 2021, and spiked between Nov 09, 2021 and Nov 11,2021 for the first time. We are not aware of the reason which caused the spike before Nov 15, 2021.
The pageviews of talk pages from mobile web logged-out users is not 0 before the official enable date. It might be due to mis-categorization of access method.
Based on the above observation, we select 2 weeks between 2021-10-24 and 2021-11-06 as the pre-deployment stage. We selected 2 weeks between 2021-11-28 and 2021-12-11 as the post-deployment stage. We estimated that the daily pageviews grew 195k due to enabling talk page to mobile web logged-out users.

jwang updated the task description. (Show Details)Jan 20 2022, 10:09 PM

Jdlrobson awarded a token.Jan 21 2022, 12:00 AM

@ovasileva, please find the raw data of above graphs in google sheet.

jwang updated the task description. (Show Details)Apr 13 2022, 9:24 PM

jwang added a project: Product-Analytics (Kanban).

jwang moved this task from Next 2 weeks to [Deprecated] Done (previously: Needs sign-off) on the Product-Analytics (Kanban) board.

jwang closed this task as Resolved.Apr 13 2022, 9:26 PM

	F34924994: image.png
	Jan 20 2022, 10:09 PM

	F34923476: image.png
	Jan 19 2022, 10:13 PM

	F34922302: image.png
	Jan 19 2022, 12:33 AM

Initial analysis of the introduction of the talk page tab for mobile anonsClosed, ResolvedPublicActions