Page MenuHomePhabricator

Identify and document success metrics for Summarization, create spec for instrumentation
Closed, ResolvedPublic

Description

Background

User story

  • As a team looking to verify hypotheses around article summarization, we would like to understand when and how the hypothesis might succeed and fail / what metrics we need to consult, so we know what next steps to take and how

Requirements

  • Identify metrics of interest
  • Document these for future reference in a specification

This task was created by Version 1.2.0 of the Web team task template using phabulous

Details

Other Assignee
jwang

Event Timeline

Jdlrobson triaged this task as Medium priority.Sep 12 2024, 4:25 PM
Jdlrobson renamed this task from Identify, document, and instrument success metrics for Figma Summarization prototype to Identify, document, and instrument success metrics for Summarization.Sep 12 2024, 5:28 PM
Jdlrobson reassigned this task from jwang to JScherer-WMF.
Jdlrobson updated Other Assignee, added: jwang.
Jdlrobson added a subscriber: jwang.

From team discussion:

  • Instrumentation will be done in statsd for both clicks to the accordion and button clicks.
  • For buttons - track thumbs up and thumbs down, no need to track unclicks

Notes from team discussion:
Proving whether users are interested:

  • Out of all pageviews where summaries are displayed, X% of pageviews had a summary opened?
  • (if possible) Out of all users that have opened a summary once who have a session that is longer than a single page, X% of them open it again on a different page

It’s useful:

  • Total number of clicks to “yes” and “no”
  • “Yes” being more than “no”

Sanity

  • How often do users who have downloaded the extension see a page where a summary is included
  • Which articles have summaries opened for most frequently
ovasileva renamed this task from Identify, document, and instrument success metrics for Summarization to Identify and document success metrics for Summarization, create spec for instrumentation.Oct 9 2024, 6:35 PM
ovasileva raised the priority of this task from Medium to High.
ovasileva updated the task description. (Show Details)

Hi @ovasileva, I have drafted an instrumentation spec for Summarization feature, based on our discussion. As we plan to instrument using statsd, which is quite different from EventLogging. I pass it onto the engineers to evaluate its feasibility.

Hi @ovasileva, I have drafted an instrumentation spec for Summarization feature, based on our discussion. As we plan to instrument using statsd, which is quite different from EventLogging. I pass it onto the engineers to evaluate its feasibility.

Thank you @jwang!

The spec has been created on this already, work for this will be handled in https://phabricator.wikimedia.org/T379214