Page MenuHomePhabricator

Real-time preview success metrics
Open, Needs TriagePublic

Description

User Story
As a product manager, I want to understand the impact of the real-time preview on edits on the platform, so that I can make an informed decision on how to deploy to users.

Acceptance Criteria

  • The metric should account for "low-quality" edits -- defined as edits with typos, or edits that get reverted. This may be difficult so any proxy metrics, even if imperfect are preferred.

Notes

  • Ideally the metric does not look at "total edit count" as a one dimensional "good thing"

Related tag for basic use metrics of the Real-time preview:
https://phabricator.wikimedia.org/T298218

Event Timeline

@NRodriguez I'm trying to understand how we measure realtime preview success. Is it a calculation from the data gathered in T298218?

To my mind, it seems like we should be looking at two primary numbers: how often RTP is used (i.e. the preview-realtime-on action), and how many edits are made. The hypothesis being that if people preview more they'll save less. Perhaps it'd be interesting to also look at the number of small edits, or those made very close together (idea being that these are more likely to be 'previewing' edits).

Given that the majority of previews currently are full-page reloads, it sort of makes sense that people don't bother with them and instead just hit publish. The preview time in both cases is the same, and if there's no further edits to make they've saved themselves a click.

Logging extra data is tracked in T306176 (but I'm not sure any of those extra bits are 100% required for a basic success metric).

To my mind, it seems like we should be looking at two primary numbers: how often RTP is used (i.e. the preview-realtime-on action), and how many edits are made. The hypothesis being that if people preview more they'll save less. Perhaps it'd be interesting to also look at the number of small edits, or those made very close together (idea being that these are more likely to be 'previewing' edits).

@Samwilson that's a good stab at a number that could be meaningful

I think if possible, it would be great to see:

Of the edits that published an edit after using real-time preview, how many did NOT get reverted?

Hi @MNeisler! We need your expertise for this task where we are trying to measure success for real-time preview.

  • What data do you think we should be looking at?
  • How should we look at the data?
  • Is there any additional data that we should track besides T298218?

Thank you again for your support with this project!

Hi @HMonroy! I've tagged the Product-Analytics on this request. We will review and prioritize the request during our next board review meeting on Tuesday, 24 May 2022.

kzimmerman added a subscriber: kzimmerman.

@HMonroy we don't have availability to take on work related to this, so I'm moving it to our tracking column.

We can help address your questions during our consultation hours; feel free to book 2 back-to-back sessions with Megan or someone else from our team if needed.