Define and instrument bounce rate on talk pages
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

Background

One of the main questions from T294503: Measuring the impact of exposing talk pages to mobile web anons is the evaluation of whether readers and other anonymous users understand how to interact with the current version of the talk page. One of the metrics we can focus on to measure this is bounce rate. This task will focus on defining the definition of bounce rate we will use for this measurement, as well as outline the instrumentation required to implement it.

Acceptance criteria

Create schema that measure interactions based on the heading: "Hypothesis: Readers will not understand what the talk page is and how to interact with it" https://docs.google.com/spreadsheets/d/11o7ZBtFff2Bi2L0kD-qC1c4XS82QqVs8fjcms824aO0/edit#gid=0

QA

Remind yourself how to check instrumentation is running.
The beta cluster should be sending events to https://intake-analytics.wikimedia.beta.wmflabs.org/v1/events?hasty=true at 100% for mobile traffic. Filter network tab to events

Visit https://en.m.wikipedia.beta.wmflabs.org/wiki/Talk:Spain and check an init event is sent.
Click article/talk page tab and check they fire events.
Click "Add new discussion" should fire an event
Clicking a section heading should fire an event
Clicking read as wiki page should fire an event

Details

Other Assignee: Jdlrobson

Subject	Repo	Branch	Lines +/-
We need some way to distinguish namespaces	mediawiki/extensions/WikimediaEvents	wmf/1.38.0-wmf.7	+2 -1
MobileWebUIActions tracks init event	mediawiki/extensions/WikimediaEvents	wmf/1.38.0-wmf.7	+27 -11
We need some way to distinguish namespaces	mediawiki/extensions/WikimediaEvents	master	+2 -1
Set sampling rate for mobile click tracking to 100% on labs	operations/mediawiki-config	master	+4 -0
MobileWebUIActions tracks init event	mediawiki/extensions/WikimediaEvents	master	+27 -11
Add `init` as a valid enum for action field	schemas/event/secondary	master	+422 -4
Instrument mobile talk page clicks	mediawiki/skins/MinervaNeue	wmf/1.38.0-wmf.7	+11 -4
Instrument mobile talk page clicks	mediawiki/skins/MinervaNeue	master	+11 -4

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T158181 Aim for workflow equivalence for MediaWiki on desktop and mobile web
Open	None	T54165 Links to talk pages in mobile view for all anonymous users
Resolved	ovasileva	T293946 Enable talk for mobile users on enwiki
Open	None	T276497 Scale DiscussionTools to all projects
Open	None	T251207 [Epic] Scale DiscussionTools to all Wikipedias
Open	None	T313976 [Release Ticket] Offer complete set of DiscussionTools features to to all wikis on mobile
Open	ovasileva	T294503 Measuring the impact of exposing talk pages to mobile web anons
Resolved	jwang	T294738 Define and instrument bounce rate on talk pages

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

ovasileva triaged this task as High priority.Nov 1 2021, 10:46 AM

ovasileva added a project: MobileFrontend (MobileFrontend and MinervaNeue architecture).

ovasileva edited projects, added Web-Team-Backlog (Kanbanana-FY-2021-22); removed Web-Team-Backlog.

ovasileva updated the task description. (Show Details)Nov 4 2021, 9:43 AM

ovasileva removed jwang as the assignee of this task.Nov 4 2021, 10:02 AM

Might be able to use MobileWebClickTracking, @Jdlrobson to add details

LGoto set the point value for this task to 3.Nov 4 2021, 5:26 PM

LGoto moved this task from Needs Analysis to Upcoming on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.

If we use MobileWebClickTracking we'd need to:

Update the WikimediaEvents extension to add an init event. This would require a slight change to the schema to the enum value to accept "init" as an action.
Update to Minerva to wire up talk pages by adding data-event-name fields to the elements that are not being logged:
add discussion button
read as wiki page link
section headings (talk page only)

Potentially we'd want to oversample talk pages as we currently sample at 10% of users but I'd suggest doing that as part of a follow up task.

Change 736933 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[schemas/event/secondary@master] Add `init` as a valid enum for action field

https://gerrit.wikimedia.org/r/736933

gerritbot added a project: Patch-For-Review.Nov 5 2021, 12:42 AM

Change 736934 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/MinervaNeue@master] Instrument mobile talk page clicks

https://gerrit.wikimedia.org/r/736934

Change 737104 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/WikimediaEvents@master] MobileWebUIActions tracks init event

https://gerrit.wikimedia.org/r/737104

Jdlrobson moved this task from Upcoming to Code Review on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.Nov 5 2021, 5:34 PM

Change 737084 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/MinervaNeue@wmf/1.38.0-wmf.7] Instrument mobile talk page clicks

https://gerrit.wikimedia.org/r/737084

LGoto assigned this task to • nray.Nov 8 2021, 6:43 PM

LGoto updated Other Assignee, added: Jdlrobson.

Jdlrobson claimed this task.Nov 8 2021, 7:04 PM

Jdlrobson moved this task from Code Review to Doing on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.

Jdlrobson updated Other Assignee, added: • nray; removed: Jdlrobson.

Jdlrobson added a subscriber: • nray.

Change 736934 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] Instrument mobile talk page clicks

https://gerrit.wikimedia.org/r/736934

Change 737084 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@wmf/1.38.0-wmf.7] Instrument mobile talk page clicks

https://gerrit.wikimedia.org/r/737084

Mentioned in SAL (#wikimedia-operations) [2021-11-08T19:41:10Z] <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.7/skins/MinervaNeue/includes/Skins/SkinMinerva.php: rSMIN8375e38ee4d5: Instrument mobile talk page clicks (T294738) (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2021-11-08T19:42:16Z] <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.7/skins/MinervaNeue/resources/: rSMIN8375e38ee4d5: Instrument mobile talk page clicks (T294738) (duration: 00m 54s)

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.7; 2021-11-02).Nov 8 2021, 8:00 PM

Jdlrobson reassigned this task from Jdlrobson to • nray.Nov 9 2021, 12:05 AM

Jdlrobson removed • nray as the assignee of this task.

Jdlrobson moved this task from Doing to Needs More Work on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.

Jdlrobson updated Other Assignee, added: Jdlrobson; removed: • nray.

Jdlrobson moved this task from Needs More Work to Code Review on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.

Change 736933 merged by jenkins-bot:

[schemas/event/secondary@master] Add `init` as a valid enum for action field

https://gerrit.wikimedia.org/r/736933

The MobileWebUIActionsTracking schema which is active for 10% of all projects currently sees 4 events a second. Introducing an init field will increase the number of events.

The WMEDesktopWebUIActionsTracking schema is active for 20% of desktop improvement projects, so doesn't include English Wikipedia. It sees around 100 events a second. It includes an init field. According to Turnilo desktop improvements projects see 70.4 m page views daily. For the same day, all projects received 711.9 m page views. 377.8 m of these were mobile (about 53% is mobile).

Therefore, enabling WMEDesktopWebUIActionsTracking for all users, should lead to 10x the number of events e.g. 1000 events a second.
Since mobile traffic is more, and touch areas more obviously, I'd expect adding an init event to lead to slightly more than 1000 events a second for Schema:MobileWebUIActionsTracking without a sampling change. This would be on par with the VirtualPageViews metric, which we are capable of serving, but we may want to lower the sampling rate (at least for English Wikipedia) a little given this is not a key metric.

SNowick_WMF subscribed.Nov 9 2021, 6:06 PM

ldelench_wmf moved this task from Triage to Tracking on the Product-Analytics board.Nov 9 2021, 6:10 PM

LGoto assigned this task to • nray.Nov 9 2021, 6:12 PM

Jdlrobson reassigned this task from • nray to Edtadros.Nov 9 2021, 9:43 PM

Jdlrobson claimed this task.

Jdlrobson moved this task from Code Review to QA on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.

Jdlrobson updated the task description. (Show Details)

Jdlrobson added a subscriber: Edtadros.

Change 737794 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[operations/mediawiki-config@master] Set sampling rate for mobile click tracking to 100% on labs

https://gerrit.wikimedia.org/r/737794

Change 737104 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] MobileWebUIActions tracks init event

https://gerrit.wikimedia.org/r/737104

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.9; 2021-11-16); removed MW-1.38-notes (1.38.0-wmf.7; 2021-11-02).Nov 9 2021, 11:00 PM

Change 737794 merged by jenkins-bot:

[operations/mediawiki-config@master] Set sampling rate for mobile click tracking to 100% on labs

https://gerrit.wikimedia.org/r/737794

Jdlrobson mentioned this in T295432: Lower sampling rate for MobileWebUIClickTracking on English Wikipedia before wmf8 is on English Wikipedia.Nov 10 2021, 1:04 AM

Jdlrobson updated the task description. (Show Details)Nov 10 2021, 1:32 AM

ovasileva mentioned this in T280051: Make mobilefrontend talk overlay disableable.Nov 10 2021, 1:09 PM

Jdlrobson reassigned this task from Jdlrobson to Edtadros.Nov 10 2021, 10:23 PM

Jdlrobson updated the task description. (Show Details)

Jdlrobson removed Edtadros as the assignee of this task.Nov 12 2021, 11:09 PM

Jdlrobson moved this task from QA to Needs More Work on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.

Jdlrobson moved this task from Needs More Work to Code Review on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.

Change 738487 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/WikimediaEvents@master] We need some way to distinguish namespaces

https://gerrit.wikimedia.org/r/738487

Change 738399 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/WikimediaEvents@wmf/1.38.0-wmf.7] MobileWebUIActions tracks init event

https://gerrit.wikimedia.org/r/738399

Change 738487 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] We need some way to distinguish namespaces

https://gerrit.wikimedia.org/r/738487

Change 739004 had a related patch set uploaded (by Nray; author: Jdlrobson):

[mediawiki/extensions/WikimediaEvents@wmf/1.38.0-wmf.7] We need some way to distinguish namespaces

https://gerrit.wikimedia.org/r/739004

Change 738399 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@wmf/1.38.0-wmf.7] MobileWebUIActions tracks init event

https://gerrit.wikimedia.org/r/738399

Change 739004 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@wmf/1.38.0-wmf.7] We need some way to distinguish namespaces

https://gerrit.wikimedia.org/r/739004

Mentioned in SAL (#wikimedia-operations) [2021-11-16T00:23:44Z] <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.7/extensions/WikimediaEvents/: 738399: 739004: WikimediaEvents backports (T294738) (duration: 00m 56s)

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.7; 2021-11-02); removed MW-1.38-notes (1.38.0-wmf.9; 2021-11-16).Nov 16 2021, 1:00 AM

LGoto assigned this task to • nray.Nov 16 2021, 6:14 PM

ovasileva mentioned this in T295915: Schema QA for measuring impact of exposing talk pages to mobile anons.Nov 17 2021, 7:09 PM

ovasileva mentioned this in T295961: Initial analysis of the introduction of the talk page tab for mobile anons.Nov 18 2021, 10:49 AM

cjming reassigned this task from • nray to Edtadros.Nov 23 2021, 2:53 PM

cjming moved this task from Code Review to QA on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.

Jdlrobson edited projects, added MobileFrontend; removed MobileFrontend (MobileFrontend and MinervaNeue architecture).Nov 30 2021, 5:58 PM

@Jdlrobson , I need some more detail around this part of the QA steps:

Remind yourself how to check instrumentation is running.
The beta cluster should be sending events to https://intake-analytics.wikimedia.beta.wmflabs.org/v1/events?hasty=true at 100% for mobile traffic. Filter network tab to events

@Edtadros the instrumentation doesn't seem to be running on the beta cluster so I think the QA steps are the right ones, they're just not actionable! Not sure why at this point. Will dig in a little deeper.

We can QA this in production now. I'll look at the data with Jennifer to verify we're seeing what's expected.

I checked this with Jennifer. Jennifer can you sign off when you are happy with this data?

@jwang - discussed this in standup today. Does it make sense to do a quick check for this task and then document the full QA in T295915: Schema QA for measuring impact of exposing talk pages to mobile anons?

discussed this in standup today. Does it make sense to do a quick check for this task and then document the full QA in T295915: Schema QA for measuring impact of exposing talk pages to mobile anons?

Sure. I am working on it now.

Tested the availability of the events for below scenarios on anonymous users:
✅ init events are available in schema

query condition:  event.action='init' and event.isanon=true

✅ events of clicking on article or talk page tab are available in schema

query condition: event.action='click'  and event.isanon=true and event.name='tabs.subject' for article tab or event.name='tabs.talk' for talk tab

✅ events of clicking "Add new discussion" are available in schema

query condition: event.action='click'  and event.isanon=true and event.name='talkpage.add-topic'

✅ events of clicking a section heading are available in schema

query condition: event.action='click'  and event.isanon=true and event.name='talkpage.section'

✅ events of clicking read as wiki page are available in schema

query condition: event.action='click'  and event.isanon=true and event.name='talkpage.readAsWiki'

In T294738#7563508, @jwang wrote:
Tested the availability of the events for below scenarios on anonymous users:
✅ init events are available in schema
query condition:  event.action='init' and event.isanon=true
✅ events of clicking on article or talk page tab are available in schema
query condition: event.action='click'  and event.isanon=true and event.name='tabs.subject' for article tab or event.name='tabs.talk' for talk tab
✅ events of clicking "Add new discussion" are available in schema
query condition: event.action='click'  and event.isanon=true and event.name='talkpage.add-topic'
✅ events of clicking a section heading are available in schema
query condition: event.action='click'  and event.isanon=true and event.name='talkpage.section'
✅ events of clicking read as wiki page are available in schema
query condition: event.action='click'  and event.isanon=true and event.name='talkpage.readAsWiki'

Looks good, resolving. Thanks @jwang!

jwang mentioned this in T297820: Schema QA instrumentation to measure bounce rate on talk pages.Dec 15 2021, 7:15 PM

ppelberg mentioned this in T301026: Create instrumentation spec for mobile DiscussionTools.Feb 23 2022, 9:23 PM

MNeisler mentioned this in T302999: Create instrumentation spec for Usability Improvements.Mar 14 2022, 4:12 PM

	ovasileva
	Nov 1 2021, 10:45 AM

Define and instrument bounce rate on talk pagesClosed, ResolvedPublic3 Estimated Story PointsActions