Page MenuHomePhabricator

Create instrumentation spec for mobile DiscussionTools
Closed, ResolvedPublic

Description

We have three analyses planned to evaluate the impact of the suite interventions we will be deploying to mobile web talk pages:

  1. T298058: [Impact Analysis] Evaluate Impact of Mobile Reply and New Discussion Tools
  2. T298065: [Impact Analysis] Evaluate Impact of mobile Topic Subscriptions + Usability Improvements
  3. T298062: [A/B Test] Run an A/B test to evaluate impact of mobile DiscussionTools

This task involves the work with defining what – if any – new instrumentation will need to be added to mobile web talk pages (both the MobileFrontend talk page overlay as well as the Read as wiki page view) in order to complete the analyses listed above.

Approaches

Potential approaches for instrumenting the mobile talk overlay and read as wiki page views in a way that enables us to compare how people use each.

  • Approach #1: Use EditAttempt to track how people use/engage with both talk page views
    • This approach is not feasible. Reason: it is too complex and inconsistent with the purpose of EditAttempt to add page view data to it.
  • Approach #2: Use MobileWebUIAction and EditAttemptStep schemas
    • Considerations
      • There are inherent disparities between the UX of the mobile frontend overlay and the read as wiki page view
  • Approach #3: Join EditAttemptStep, Talk_Page_Edit, and MobileWebUIActions
    • Considerations
      • Doing the above would require that the sampling rates these three schemas use and the sessionIDs they generate could be made consistent. Note: investigations of the feasibility of this are happening in T303653 and T303654.

Open question(s)

  • 1. What – if any – new instrumentation will need to be added to mobile web talk pages in order to complete the analyses listed below?
  • T298058
  • T298065
  • T298062
NOTE: the Web Team has existing instrumentation in place that we might be able to reuse to meet our needs. See T295490 and T294738.

Instrumentation Spec

Instrumentation Spec

Done

  • Answers to all ===Open question(s) are documented in this ticket
  • Instrumentation spec is created that documents all of the events we will need in order to report on the metrics defined in the Talk Pages Project/Mobile/Measurement Plan

Related Objects

Event Timeline

To clarify, is this about the mobile talk overlay, or just when you view the desktop talk page from a mobile device?

To clarify, is this about the mobile talk overlay, or just when you view the desktop talk page from a mobile device?

I'm glad you thought to ask, @DLynch. Does the below resolve the ambiguity you noticed and/or bring new thoughts/questions to mind?

This ticket is about determining what – if any – new instrumentation we'll need to add to the three versions of mobile talk pages [i] that will exist once all DiscussionTools features become available on mobile. Note: this instrumentation is all in service of completing the analyses we are planning in T298072.

Talk Page Versions
DescriptionScreenshot
The MobileFrontend talk page overlay
IMG_80B18E498BFF-1.jpeg (2×1 px, 592 KB)
The Read as wiki page view
IMG_F3761F027884-1.jpeg (2×1 px, 466 KB)
The Read as wiki page view with the suite of mobile DiscussionTools enabled (T278588)TBD

i. Maybe there will technically be two NOT three versions of mobile web talk pages?

ppelberg renamed this task from [SPIKE] Determine what – if any – new instrumentation is required for mobile web talk pages to Create instrumentation space for mobile DiscussionTools.Feb 23 2022, 9:36 PM
ppelberg assigned this task to MNeisler.
ppelberg updated the task description. (Show Details)
ppelberg moved this task from Backlog to Analytics on the Editing-team (Tracking) board.
ppelberg updated the task description. (Show Details)
ppelberg added a subscriber: MNeisler.
MNeisler renamed this task from Create instrumentation space for mobile DiscussionTools to Create instrumentation spec for mobile DiscussionTools.Feb 25 2022, 12:09 PM
MNeisler triaged this task as Medium priority.
MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.

@ppelberg

Here is the current draft instrumentation spec, where I've taken an initial pass at reviewing and defining all the instrumentation that will be needed to measure the metrics identified to evaluate DiscussionTools on Mobile (as described in the measurement plan).

The spec includes columns that identify events to be tracked, whether they are currently instrumented in either of the two possible talk page versions (Read as Wiki Page or Mobile Front End) and where the existing instrumentation is implemented.

Summary of findings from the review :

  • MobileWebUIActions: The MobileWebUIActions schema currently tracks clicks to start a new discussion and views to a talk page on mobile, which are events we need to track for the planned analyses. If we decide to use this schema for our planned analyses, we will need to resolve the following:
    • Resolve T295490 to make the click tracking that is present in the MobileFrontend talk page overlay within the "Read as wiki page" view.
    • This schema does not appear to log clicks to start a reply or clicks to start a page or section edit, which are identified as forms of talk page engagement in our measurement plan. This instrumentation should be added if we plan to use this schema to measure bounce rate.
    • This schema does not include a unique identifier (only a unique session id). We would need to look into adding a unique identifier with this schema if we want to look at per user metrics vs per session metrics.
  • EditAttemptStep and Talk_Page_Edit:
    • No data is being tracked in the current mobile frontend talk experience. We would need to add any relevant events to the mobile talk overlay actions. Column B of the spec identifies the key talk page events that we want to track. Mapping these events to the actions people can take within the mobile front end talk experience will be completed as part of T302108.
    • There is an unresolved bug related to EAS recording all Wikiedtior edits as desktop edits. See T249944. Need to confirm the extent of impact of this (if there is any) on the planned analyses for this project.

On 10 March (yesterday), @DLynch, @MNeisler, and I met to discuss the findings Megan posted in T301026#7765304 and the questions they brought about so that we can decide how to move forward with instrumenting the mobile talk overlay and read as wiki page views.

The action items that emerged during this meeting are documented below. I've also added the ===Approaches section to the task description which now contains the three instrumentation approaches we discussed during yesterday's meeting.

10 March Meeting Outcomes

Issue/QuestionTicket
Investigate consistency of sessionID across schemasT303653
Determine what would be required to ensure sessionID is consistent across schemasT303653
Determine what would be required to ensure sampling rate is consistent across schemasT303654

@ppelberg This has been in the needs sign-off column on the product analytics kanban since March. Is there work left to do here for @MNeisler?

@ppelberg This has been in the needs sign-off column on the product analytics kanban since March. Is there work left to do here for @MNeisler?

hi @mpopov! There could be more work for @MNeisler. Tho, I don't think we will know whether there is more work to be done or not until after T303654 is resolved.

@MNeisler and I will talk about whether more work is needed on this ticket the next time we meet.

Per today's discussion with @MNeisler, the instrumentation required to complete the analyses named in the task description (T298058, T298065, and T298062) is complete.