Page MenuHomePhabricator

Make click tracking work on reply tool talk page
Closed, ResolvedPublic

Description

We use click tracking to measure bounce rate however when disabling talk pages as on https://en.m.wikipedia.beta.wmflabs.org/wiki/Talk:Spain several key HTML markup is lost that prevents us tracking clicks to key components. These are the add discussion button and section headings.

Screen Shot 2021-11-10 at 8.44.15 AM.png (218×700 px, 23 KB)

Screen Shot 2021-11-10 at 8.44.59 AM.png (182×2 px, 17 KB)

For click tracking to work, the clickable item needs to have a data-event-name attribute.

Open question(s)

  • 1. Web Team: in what schema would you value the clicks/taps on the section headings that appear within the mobile "Read as wiki page" talk page view be logged within?

Done

  • A) Clicks should be recorded when add discussion is clicked (data-event-name="talkpage.add-topic")
  • B) Clicks should be recorded when headings are clicked (data-event-name="talkpage.section")
  • Editing Team QTE verifies clients are emitting events as expected
  • Product Analytics verifies client-side events are being logged as expected on the server

Developer notes

The data-event-name attribute can be set in HTML or in JS eg..
https://github.com/wikimedia/mediawiki-skins-MinervaNeue/blob/master/resources/skins.minerva.scripts/talk.js#L113

Related Objects

Event Timeline

Restricted Application added a subscriber: Masumrezarock100. · View Herald Transcript

Change 738012 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@master] Split DiscussionToolsEnableMobile=true into 'behind-overlay' and 'remove-overlay'

https://gerrit.wikimedia.org/r/738012

Change 738012 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] Split DiscussionToolsEnableMobile=true into 'behind-overlay' and 'remove-overlay'

https://gerrit.wikimedia.org/r/738012

@Jdlrobson: would it be accurate for the Editing Team to think that we do NOT need to prioritize implementing the click tracking that is present in the MobileFrontend talk page overlay within the "Read as wiki page" view?

Thinking: the MobileFrontend talk page overlay, and the instrumentation that's been implemented within it, will NOT be removed until the Web Team has completed the analysis (T294503) that depends on said instrumentation/click tracking. More context in
T295816#7561919.

I think that's a product decision @ppelberg and @ovasileva and not a technical decision?

It seems like it would be useful to be able to compare bounce rates before and after turning on structured discussions to understand the impact of the product change on anons and logged-in users by switching to the "read as wiki page" view. From a technical point of view, the bounce rate instrumentation is pretty cheap to wire up on the new implementation if you need it (a data attribute on the clickable element). Happy to walk you through that code if it's not clear.

FWIW, Right now, for December I see 5,411 session clicks to the read as wiki page button [1] out of a total of 41,554 talk page views [2]. I see 10,511 [6] section heading clicks. When I look at logged-in traffic only, I see 2,347 "read as wiki" clicks [3] in 4,215 sessions [4]. In contrast section headings get 3,581 clicks [6] (e.g. more people are clicking sections than "read as wiki page".
e.g. It seems anonymous users are less likely to click read as wiki than logged in users. At least 85% of logged in user sessions engage with section collapsing in some form. Will be interesting to see Jennifer Wang's analysis to get a better picture - perhaps a decision around this should be blocked on those learnings?

[1] (select count( distinct event.token) from mobilewebuiactionstracking WHERE year = 2021 AND month = 12 AND event.action = 'click' AND event.name = 'talkpage.readAsWiki')
[2] select count(distinct event.token) from mobilewebuiactionstracking WHERE year = 2021 AND month = 12 AND event.action = 'init' AND event.name = "ns=1"
[3] select count( distinct event.token) from mobilewebuiactionstracking WHERE year = 2021 AND month = 12 AND event.action = 'click' AND event.name = 'talkpage.readAsWiki' AND event.isAnon = false
[4]
select count(distinct event.token) from mobilewebuiactionstracking WHERE year = 2021 AND month = 12 AND event.action = 'init' AND event.name = "ns=1" AND event.isAnon = false
[5] select count( distinct event.token) from mobilewebuiactionstracking WHERE year = 2021 AND month = 12 AND event.action = 'click' AND event.name = 'talkpage.section'
[5] select count( distinct event.token) from mobilewebuiactionstracking WHERE year = 2021 AND month = 12 AND event.action = 'click' AND event.name = 'talkpage.section' AND event.isAnon = false

I think that's a product decision @ppelberg and @ovasileva and not a technical decision?

It seems like it would be useful to be able to compare bounce rates before and after turning on structured discussions to understand the impact of the product change on anons and logged-in users by switching to the "read as wiki page" view. From a technical point of view, the bounce rate instrumentation is pretty cheap to wire up on the new implementation if you need it (a data attribute on the clickable element). Happy to walk you through that code if it's not clear.

This is helpful context – thank you for sharing it, @Jdlrobson.

We agree with you in thinking measuring the bounce rate before and after the suite of mobile DiscussionTools features are made available will be be helpful to evaluating their impact. Note: this analysis will happen in T298062.

As such, the Editing Team is going to prioritize ensuring the click tracking y'all have implemented is present within the new view and tools we will be introducing. Note: the work to decide what – if any – *additional* instrumentation will be required will happen in T301026.

A couple of clarifying questions

  • Is the data you posted in T295490#7568307 specific to any particular wiki(s)/project(s) or an aggregate count of mobile web talk page engagement across all Wikipedias?
  • Would it be accurate for me to understand the "section heading clicks" you are referring to as clicks on the discussion titles within the MobileFrontend talk page "overlay" (pictured below[i])?

...perhaps a decision around this should be blocked on those learnings?


i. MobileFrontend talk page overlay
IMG_80B18E498BFF-1.jpeg (2×1 px, 592 KB)

To opt something into click tracking all you need to do is add a data-event-name attribute to the element.

Is the data you posted in T295490#7568307 specific to any particular wiki(s)/project(s) or an aggregate count of mobile web talk page engagement across all Wikipedias?

Currently, the mobile click tracking data is collected across all wikis at a sampling rate of 1% for English Wikipedia and 10% for all other wikis.

Would it be accurate for me to understand the "section heading clicks" you are referring to as clicks on the discussion titles within the MobileFrontend talk page "overlay" (pictured below[i])?

Yes. We currently do this by adding data-event-name="talkpage.section" to all headings on the talk page.

ppelberg renamed this task from Make click tracking work on reply tool talk page to Make click tracking work "Read as wiki page" talk page view.May 19 2022, 6:43 PM

Per what @DLynch, @MNeisler, and I talked about on 18 May 2022:

  • DECIDED: the Editing Team will add the instrumentation required to the Add topic button that appears in the Read as wiki page mobile talk page view within the EditAttemptStep schema.
  • OPEN: in what schema would the Web Team value the clicks/taps on section headings being logged within? Note: taps/clicks on section headings is outside the scope of EditAttemptStep and I didn't see a particular schema named within the Talk.js file linked above.

(I've updated the task description to include the above)

EditAttemptStep

I'm pretty sure none of this should be going into that schema. Adding the data-event-name attribute should, as I understand what Jon said, make this all get logged into the MobileWebUIActions schema.

I guess if we're asking questions, we might want to make sure that these being logged with the same event-names as the existing overlay uses is going to be okay, or whether giving it something distinct would help avoid analysis-confusion.

I can confirm MobileWebUIActions schema is the one we want this to show up in.

Change 800450 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@master] Add attributes for Minerva click tracking

https://gerrit.wikimedia.org/r/800450

matmarex renamed this task from Make click tracking work "Read as wiki page" talk page view to Make click tracking work on reply tool talk page.May 27 2022, 2:02 AM
matmarex claimed this task.
matmarex updated the task description. (Show Details)

I've undone some changes to the description, the additional details were incorrect.

Change 800809 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/MobileFrontend@master] Toggler: Update icon in-place instead of re-creating it

https://gerrit.wikimedia.org/r/800809

Change 800809 merged by jenkins-bot:

[mediawiki/extensions/MobileFrontend@master] Toggler: Update icon in-place instead of re-creating it

https://gerrit.wikimedia.org/r/800809

Change 800450 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] Add attributes for Minerva click tracking

https://gerrit.wikimedia.org/r/800450

@matmarex , how do I verify that the client emits events as expected?

Screenshot 2022-06-10 at 22.35.16.png (170×1 px, 48 KB)
The is the closest thing to events that I see. Help me with steps to test this.

It looks like the events are sampled at 1% or 10% rate in production (depending on the wiki, configured using 'wgWMEMobileWebUIActionsTracking'), which means that usually they're not actually being emitted, so it's probably impractical to test this in production.

You can test it locally by installing the WikimediaEvents extension, then setting $wgWMEMobileWebUIActionsTracking = 1.0; in your LocalSettings.php. (There might be some other config needed, I don't remember right now… Let me know if you run into issues and I'll try to help.)

If the events were emitted, they would show up in the developer tools where you looked! There are also some more convenient ways to see them, documented at https://www.mediawiki.org/wiki/Extension:EventLogging/Programming#Monitoring_events.

It looks like the events are sampled at 1% or 10% rate in production (depending on the wiki, configured using 'wgWMEMobileWebUIActionsTracking'), which means that usually they're not actually being emitted, so it's probably impractical to test this in production.

You can test it locally by installing the WikimediaEvents extension, then setting $wgWMEMobileWebUIActionsTracking = 1.0; in your LocalSettings.php. (There might be some other config needed, I don't remember right now… Let me know if you run into issues and I'll try to help.)

If the events were emitted, they would show up in the developer tools where you looked! There are also some more convenient ways to see them, documented at https://www.mediawiki.org/wiki/Extension:EventLogging/Programming#Monitoring_events.

@matmarex: Checked this locally, both the events for click tracking are being emitted.

When the "Add discussion" button is clicked:

Screen Shot 2022-06-22 at 7.00.35 PM.jpeg (673×972 px, 108 KB)

When headings are clicked:

Screen Shot 2022-06-22 at 7.01.32 PM.jpeg (672×979 px, 125 KB)

One question though, we are only tracking when the "Add discussion" button is clicked but not when the "Start a discussion" button is clicked on a user's talk page, correct?

Screen Shot 2022-06-22 at 7.03.18 PM.jpeg (545×967 px, 91 KB)

One question though, we are only tracking when the "Add discussion" button is clicked but not when the "Start a discussion" button is clicked on a user's talk page, correct?

Yes, I think that's OK. It's a different button and I don't think we need that to match the old logging. We track opening the new topic tool through that button in our own schema.

One question though, we are only tracking when the "Add discussion" button is clicked but not when the "Start a discussion" button is clicked on a user's talk page, correct?

Yes, I think that's OK. It's a different button and I don't think we need that to match the old logging. We track opening the new topic tool through that button in our own schema.

Right, alright we can close this one then.

I checked the aggregate data in MobileWebUIActionTracking and confirmed that these new events are logging are expected.

To query the data, I looked for sessions that included clicks to the read as wiki page button on talk pages (event.action = 'click' AND event.name = 'talkpage.readAsWiki') and subsequent clicks to event.name = 'talkpage.add-topic' or event.name = 'talkpage.section'.