Page MenuHomePhabricator

Create instrumentation spec for Usability Improvements
Closed, ResolvedPublic

Description

We have four quantitative analyses planned to evaluate the impact of the Usability Improvements the Editing Team is working on (T249579):

Desktop AnalysesMobile Analyses
T302359 + T302358T298065 + T298062

This task involves the work with defining what – if any – new instrumentation will need to be added to talk pages (mobile (both the MobileFrontend talk page overlay as well as the Read as wiki page view) and desktop) in order to complete the analyses listed above.

Open question(s)

  • 1. What – if any – new instrumentation will need to be added to mobile web talk pages in order to complete the analyses listed below?
  • T302359
  • T302358
  • T298065
  • T298062

Instrumentation Spec

UsabilityImprovements/InstrumentationSpec

Done

Related Objects

Event Timeline

MNeisler triaged this task as Medium priority.Mar 8 2022, 5:29 PM
MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.

This task involves the work with defining what – if any – new instrumentation will need to be added to talk pages (mobile (both the MobileFrontend talk page overlay as well as the Read as wiki page view) and desktop) in order to complete the analyses listed above.

@ppelberg - I thought that the Usability Improvement analyses identified in this task (T302359 and T302358) were focused on the deployment of this feature to desktop, while analysis of Usability Improvements on mobile would be conducted as part of the planned analyses for the DiscussionTools on Mobile project (T298065 and T298062). Can you confirm/clarify? Thanks!

I've completed the instrumentation review and spec for the Usability Improvements projects, which defines all the instrumentation that will be needed to measure the metrics in the measurement plan.

Based on my review, additional instrumentation is needed to track desktop talk page views associated with sessions where the user engages on a talk page (defined as clicks an affordance to start a new discussion/reply, subscribes, or edits). See details below. I've confirmed all other events we need to track for this project are already instrumented in EditAttemptStep, talk_page_edit, and mediawiki_history.

NEEDED INSTRUMENTATION

  • Event: A user views a talk page on desktop
  • Needed for the following metrics identifed in the measurement plan: "Increase in the proportion of Junior Contributors who visit a talk page and engage with it." ; "Of all the Junior or Senior Contributors that post on a talk page, the average time from when a Senior Contributor views a talk page to starting any new edit on a talk page"; * "The percentage of Junior Contributors that view a talk page and successfully publish at least one comment or new discussion"
  • Instrumentation option: We can use the DesktopWebUIActions schema, which currently tracks desktop page views (recorded as `action = 'init'). To use this schema, we would need to confirm the following [i] : (1) that the events in DesktopWebUIActions can be joined with EditAttemptStep and Talk_page_edit events using the session_id and (2) we can synchronize the sampling rates and methods so the same events are sampled. [ii]

[i]This is related to work being done in T303653 and T303654 except focused on desktop pageviews vs mobile web pageviews.
[ii] Sampling rate considerations: Per T294738#7492873, DesktopWebUIActionsTracking schema is only active for desktop improvement projects, so doesn't include English Wikipedia and DesktopWebUIActionsTracking schema is currently sampled with a rate of 20% for logged out users, and a rate of 100% for logged in users (per T287545#7449966)

This task involves the work with defining what – if any – new instrumentation will need to be added to talk pages (mobile (both the MobileFrontend talk page overlay as well as the Read as wiki page view) and desktop) in order to complete the analyses listed above.

@ppelberg - I thought that the Usability Improvement analyses identified in this task (T302359 and T302358) were focused on the deployment of this feature to desktop, while analysis of Usability Improvements on mobile would be conducted as part of the planned analyses for the DiscussionTools on Mobile project (T298065 and T298062). Can you confirm/clarify? Thanks!

Great spot, @MNeisler. What you described above is accurate:

  • Desktop: we have two analyses planned to evaluate the Usability Improvements that will be made available on desktop (T302359 and T302358)
  • Mobile: we have two separate analyses planned to evaluate the Usability Improvements that will be made available on mobile (TT298065 and T298062)

I've updated the task description to reflect the above.