Page MenuHomePhabricator

Define and instrument bounce rate on talk pages
Open, HighPublic3 Estimated Story Points

Description

Background

One of the main questions from T294503: [EPIC] Measuring the impact of exposing talk pages to mobile web anons is the evaluation of whether readers and other anonymous users understand how to interact with the current version of the talk page. One of the metrics we can focus on to measure this is bounce rate. This task will focus on defining the definition of bounce rate we will use for this measurement, as well as outline the instrumentation required to implement it.

Acceptance criteria

QA

Remind yourself how to check instrumentation is running.
The beta cluster should be sending events to https://intake-analytics.wikimedia.beta.wmflabs.org/v1/events?hasty=true at 100% for mobile traffic. Filter network tab to events

  • Visit https://en.m.wikipedia.beta.wmflabs.org/wiki/Talk:Spain and check an init event is sent.
  • Click article/talk page tab and check they fire events.
  • Click "Add new discussion" should fire an event
  • Clicking a section heading should fire an event
  • Clicking read as wiki page should fire an event

Event Timeline

Might be able to use MobileWebClickTracking, @Jdlrobson to add details

LGoto set the point value for this task to 3.Thu, Nov 4, 5:26 PM

If we use MobileWebClickTracking we'd need to:

  • Update the WikimediaEvents extension to add an init event. This would require a slight change to the schema to the enum value to accept "init" as an action.
  • Update to Minerva to wire up talk pages by adding data-event-name fields to the elements that are not being logged:
  • add discussion button
  • read as wiki page link
  • section headings (talk page only)

Potentially we'd want to oversample talk pages as we currently sample at 10% of users but I'd suggest doing that as part of a follow up task.

Change 736933 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[schemas/event/secondary@master] Add `init` as a valid enum for action field

https://gerrit.wikimedia.org/r/736933

Change 736934 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/MinervaNeue@master] Instrument mobile talk page clicks

https://gerrit.wikimedia.org/r/736934

Change 737104 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/WikimediaEvents@master] MobileWebUIActions tracks init event

https://gerrit.wikimedia.org/r/737104

Change 737084 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/MinervaNeue@wmf/1.38.0-wmf.7] Instrument mobile talk page clicks

https://gerrit.wikimedia.org/r/737084

LGoto updated Other Assignee, added: Jdlrobson.
Jdlrobson updated Other Assignee, added: nray; removed: Jdlrobson.
Jdlrobson added a subscriber: nray.

Change 736934 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@master] Instrument mobile talk page clicks

https://gerrit.wikimedia.org/r/736934

Change 737084 merged by jenkins-bot:

[mediawiki/skins/MinervaNeue@wmf/1.38.0-wmf.7] Instrument mobile talk page clicks

https://gerrit.wikimedia.org/r/737084

Mentioned in SAL (#wikimedia-operations) [2021-11-08T19:41:10Z] <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.7/skins/MinervaNeue/includes/Skins/SkinMinerva.php: 8375e38ee4d57b4ff3d30be96473b927ac8e4ef0: Instrument mobile talk page clicks (T294738) (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2021-11-08T19:42:16Z] <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.7/skins/MinervaNeue/resources/: 8375e38ee4d57b4ff3d30be96473b927ac8e4ef0: Instrument mobile talk page clicks (T294738) (duration: 00m 54s)

Jdlrobson removed nray as the assignee of this task.
Jdlrobson updated Other Assignee, added: Jdlrobson; removed: nray.

Change 736933 merged by jenkins-bot:

[schemas/event/secondary@master] Add `init` as a valid enum for action field

https://gerrit.wikimedia.org/r/736933

The MobileWebUIActionsTracking schema which is active for 10% of all projects currently sees 4 events a second. Introducing an init field will increase the number of events.

The WMEDesktopWebUIActionsTracking schema is active for 20% of desktop improvement projects, so doesn't include English Wikipedia. It sees around 100 events a second. It includes an init field. According to Turnilo desktop improvements projects see 70.4 m page views daily. For the same day, all projects received 711.9 m page views. 377.8 m of these were mobile (about 53% is mobile).

Therefore, enabling WMEDesktopWebUIActionsTracking for all users, should lead to 10x the number of events e.g. 1000 events a second.
Since mobile traffic is more, and touch areas more obviously, I'd expect adding an init event to lead to slightly more than 1000 events a second for Schema:MobileWebUIActionsTracking without a sampling change. This would be on par with the VirtualPageViews metric, which we are capable of serving, but we may want to lower the sampling rate (at least for English Wikipedia) a little given this is not a key metric.

Jdlrobson claimed this task.
Jdlrobson updated the task description. (Show Details)
Jdlrobson added a subscriber: Edtadros.

Change 737794 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[operations/mediawiki-config@master] Set sampling rate for mobile click tracking to 100% on labs

https://gerrit.wikimedia.org/r/737794

Change 737104 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] MobileWebUIActions tracks init event

https://gerrit.wikimedia.org/r/737104

Change 737794 merged by jenkins-bot:

[operations/mediawiki-config@master] Set sampling rate for mobile click tracking to 100% on labs

https://gerrit.wikimedia.org/r/737794

Jdlrobson updated the task description. (Show Details)
Jdlrobson updated the task description. (Show Details)

Change 738487 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/WikimediaEvents@master] We need some way to distinguish namespaces

https://gerrit.wikimedia.org/r/738487

Change 738399 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/extensions/WikimediaEvents@wmf/1.38.0-wmf.7] MobileWebUIActions tracks init event

https://gerrit.wikimedia.org/r/738399

Change 738487 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] We need some way to distinguish namespaces

https://gerrit.wikimedia.org/r/738487

Change 739004 had a related patch set uploaded (by Nray; author: Jdlrobson):

[mediawiki/extensions/WikimediaEvents@wmf/1.38.0-wmf.7] We need some way to distinguish namespaces

https://gerrit.wikimedia.org/r/739004

Change 738399 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@wmf/1.38.0-wmf.7] MobileWebUIActions tracks init event

https://gerrit.wikimedia.org/r/738399

Change 739004 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@wmf/1.38.0-wmf.7] We need some way to distinguish namespaces

https://gerrit.wikimedia.org/r/739004

Mentioned in SAL (#wikimedia-operations) [2021-11-16T00:23:44Z] <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.7/extensions/WikimediaEvents/: 738399: 739004: WikimediaEvents backports (T294738) (duration: 00m 56s)

@Jdlrobson , I need some more detail around this part of the QA steps:

Remind yourself how to check instrumentation is running.
The beta cluster should be sending events to https://intake-analytics.wikimedia.beta.wmflabs.org/v1/events?hasty=true at 100% for mobile traffic. Filter network tab to events

@Edtadros the instrumentation doesn't seem to be running on the beta cluster so I think the QA steps are the right ones, they're just not actionable! Not sure why at this point. Will dig in a little deeper.

We can QA this in production now. I'll look at the data with Jennifer to verify we're seeing what's expected.