Page MenuHomePhabricator

Event tracking for Realtime Preview
Open, Needs TriagePublic

Description

CONTEXT

This ticket fleshes out the set of data-driven questions that can help us to understand both how to:

  • Measure the new tool's impact on the edit metrics ecosystem
  • Optimize tool design

DATA QUESTIONS

  • How often are users opening and closing the Preview window inside the Wikitext code editor?
  • How often are users being defaulted to the Manual Reload experience rather than the auto-load experience?
  • Which method are users opting for previewing their edits, shortcut or button?
  • What's the average load time for a refresh?
  • How often are users encountering the error view?

Downstream impact: Analysis that can come after

  • How many people successfully publish an unreverted edit after they use realtime preview?

IMPLEMENTATION PLAN TO BE FLESHED OUT BASED ON QUESTIONS

To come <- Task can be edited by engineer

New Instrumentation

Proposed new event nameAction that will trigger new eventSchema where event will be logged
preview-realtime-stoppedTriggered when realtime preview auto-load is disabled. Currently, this happens when the api does not repond within 6 seconds 3 consecutive times.VisualEditorFeatureUse
preview-realtime-reload-clickTriggered when reload button in the realtime preview pane is clickedVisualEditorFeatureUse
preview-realtime-reload-accesskeyTriggered when the reload button in the realtime preview pane is fired with the accesskey [Alt+Shit+)]VisualEditorFeatureUse

Implementation Process

StepPeople/Team ResponsibleDescriptionStatus
1.Community-TechFile ticket requesting instrumentationβœ… Done in T306176
2.Community-TechPopulate the ===New instrumentation section with proposed changes, once definedβœ… Done
3.@MNeisler + @DLynchReview the ===New instrumentation Community Tech is proposing to introduceβœ… Done
4.Community-TechImplement the instrumentation changes defined in ===New instrumentation
5.Editing-teamCode review the instrumentation changes Community Tech will have implemented
6.Community-TechVerify newly instrumented events are being emitted as expected by clients
7.Product-AnalyticsVerify newly instrumented events are being logged in database as expected, once the new instrumentation has landed in production.
8.Community-TechUpdate schema data dictionary with the events spec'd in this task once they have been verified to be implemented as expected. Read: clients are emitting events as expected and said events are being logged in the database as expected as well.

Event Timeline

How often are users opening and closing the Preview window inside the Wikitext code editor?

We can use the existing preview-realtime-on and preview-realtime-off event actions for this.

How often are users being defaulted to the Manual Reload experience rather than the auto-load experience?

For this we need a new action. I propose preview-realtime-stopped.

Which method are users opting for previewing their edits, shortcut or button?

Two new actions required: preview-realtime-reload-click and preview-realtime-reload-accesskey. (We are able to determine which user interaction is used to fire the reload.)

What's the average load time for a refresh?

I'm not sure how a time duration value should be stored. Perhaps there's an existing system for this sort of thing, or perhaps we can extract this info from other existing logs (e.g. the parse API? although of course that'd include other usages).

How often are users encountering the error view?

Errors are already logged, so we should be able to extract their frequency from existing logs.


So two questions for Analytics: Is it okay to add these three new actions? And how should we store the preview-load duration times?

So two questions for Analytics: Is it okay to add these three new actions? And how should we store the preview-load duration times?

CCing @MNeisler

Let us know if I can do anything to get this in the greenlight and start tracking this data!

Thanks so much for your continued support throughout this product! <3

I'm not sure how a time duration value should be stored. Perhaps there's an existing system for this sort of thing, or perhaps we can extract this info from other existing logs (e.g. the parse API? although of course that'd include other usages).

There's logging into statsd that you might use for this. mw.track( 'timing.foo.bar', 750 ) should get logged to statsd as foo.bar for 750ms.

How often are users opening and closing the Preview window inside the Wikitext code editor?

We can use the existing preview-realtime-on and preview-realtime-off event actions for this.

How often are users being defaulted to the Manual Reload experience rather than the auto-load experience?

For this we need a new action. I propose preview-realtime-stopped.

Which method are users opting for previewing their edits, shortcut or button?

Two new actions required: preview-realtime-reload-click and preview-realtime-reload-accesskey. (We are able to determine which user interaction is used to fire the reload.)

What's the average load time for a refresh?

I'm not sure how a time duration value should be stored. Perhaps there's an existing system for this sort of thing, or perhaps we can extract this info from other existing logs (e.g. the parse API? although of course that'd include other usages).

How often are users encountering the error view?

Errors are already logged, so we should be able to extract their frequency from existing logs.


So two questions for Analytics: Is it okay to add these three new actions? And how should we store the preview-load duration times?

@MNeisler Per our conversation, is okay to add the three new actions right? I just want to confirm πŸ˜„

@HMonroy: Thanks for checking! The proposed names for the new actions (preview-realtime-stopped, preview-realtime-reload-click, and preview-realtime-reload-accesskey) look fine to me.

Before proceeding, @ppelberg will follow up with details on the process to have these new events added to VisualEditorFeatureUse.

@HMonroy: Thanks for checking! The proposed names for the new actions (preview-realtime-stopped, preview-realtime-reload-click, and preview-realtime-reload-accesskey) look fine to me.

Before proceeding, @ppelberg will follow up with details on the process to have these new events added to VisualEditorFeatureUse.

Thank you for the ping, @MNeisler. hi @HMonroy! I've updated the task description with the steps Megan alluded to above. Please let us know if anything is unclear/unexpected/etc.

Hi @ppelberg ! I've updated the ===New instrumentation section and it's ready for you and @MNeisler to review. Please let us know when we can proceed with implementation. Thank you for guiding us through this process :)

@HMonroy Thanks for updating the ===New instrumentation section with the details.

As commented above, the proposed names look fine to me. These will all fall under event.feature = 'preview' correct?

Copying @DLynch to review as well and then you can go ahead and proceed with implementation.

Also, a reminder to please make sure to update the VEFU Data Dictionary once these are added (I added a link to this page in the Process Steps in the task description as well)

@MNeisler Correct, they will be under event.feature='preview'. They would follow the same logic as other Realtime Preview events logging: https://gerrit.wikimedia.org/g/mediawiki/extensions/WikiEditor/+/6b1698d40822f72a450a9f3b54e41b676d9e42e0/modules/ext.wikiEditor.js#262

Hi @DLynch! Please let us know if everything looks good to proceed :)

All looks fine to me. VisualEditorFeatureUse is very freeform; so long as you're avoiding collisions on the feature field and at least halfheartedly trying to match your action up with other semantically-equivalent ones, it's all good.

Decision: For the "Reload" tracking avenues, it's ok to track under one event rather than three different ways (2 buttons, and Keyboard shortcut)

(Posting notes from synch with Harumi, and the conversation about complexity with tracking granular events)