In T306176#7898379, @Samwilson wrote:

How often are users opening and closing the Preview window inside the Wikitext code editor?

We can use the existing preview-realtime-on and preview-realtime-off event actions for this.

How often are users being defaulted to the Manual Reload experience rather than the auto-load experience?

For this we need a new action. I propose preview-realtime-stopped.

Which method are users opting for previewing their edits, shortcut or button?

Two new actions required: preview-realtime-reload-click and preview-realtime-reload-accesskey. (We are able to determine which user interaction is used to fire the reload.)

What's the average load time for a refresh?

I'm not sure how a time duration value should be stored. Perhaps there's an existing system for this sort of thing, or perhaps we can extract this info from other existing logs (e.g. the parse API? although of course that'd include other usages).

How often are users encountering the error view?

Errors are already logged, so we should be able to extract their frequency from existing logs.

---

So two questions for Analytics: Is it okay to add these three new actions? And how should we store the preview-load duration times?

@MNeisler Per our conversation, is okay to add the three new actions right? I just want to confirm 😄

@HMonroy: Thanks for checking! The proposed names for the new actions (preview-realtime-stopped, preview-realtime-reload-click, and preview-realtime-reload-accesskey) look fine to me.

Before proceeding, @ppelberg will follow up with details on the process to have these new events added to VisualEditorFeatureUse.

In T306176#7992868, @MNeisler wrote:

@HMonroy: Thanks for checking! The proposed names for the new actions (preview-realtime-stopped, preview-realtime-reload-click, and preview-realtime-reload-accesskey) look fine to me.

Before proceeding, @ppelberg will follow up with details on the process to have these new events added to VisualEditorFeatureUse.

Thank you for the ping, @MNeisler. hi @HMonroy! I've updated the task description with the steps Megan alluded to above. Please let us know if anything is unclear/unexpected/etc.

Hi @ppelberg ! I've updated the ===New instrumentation section and it's ready for you and @MNeisler to review. Please let us know when we can proceed with implementation. Thank you for guiding us through this process :)

@HMonroy Thanks for updating the ===New instrumentation section with the details.

As commented above, the proposed names look fine to me. These will all fall under event.feature = 'preview' correct?

Copying @DLynch to review as well and then you can go ahead and proceed with implementation.

Also, a reminder to please make sure to update the VEFU Data Dictionary once these are added (I added a link to this page in the Process Steps in the task description as well)

@MNeisler Correct, they will be under event.feature='preview'. They would follow the same logic as other Realtime Preview events logging: https://gerrit.wikimedia.org/g/mediawiki/extensions/WikiEditor/+/6b1698d40822f72a450a9f3b54e41b676d9e42e0/modules/ext.wikiEditor.js#262

Hi @DLynch! Please let us know if everything looks good to proceed :)

All looks fine to me. VisualEditorFeatureUse is very freeform; so long as you're avoiding collisions on the feature field and at least halfheartedly trying to match your action up with other semantically-equivalent ones, it's all good.

Decision: For the "Reload" tracking avenues, it's ok to track under one event rather than three different ways (2 buttons, and Keyboard shortcut)

(Posting notes from synch with Harumi, and the conversation about complexity with tracking granular events)

@ppelberg @MNeisler @DLynch tracking the click via the access key was going to get too complicated so I reviewed this with @NRodriguez, and it was decided that it was not worth the effort. The reload button shows in realtime preview on 3 different cases. It shows on hovering, when manual reload is enabled and when there is an error. We would like to track those three different clicks.

I revised the New Instrumentation section. Please let me know if this makes sense. Thank you.

@DLynch gentle nudge. Have you had a chance to take a look at the latest New Instrumentation?

@HMonroy Oops, sorry. I got sidetracked by the whole inspiration week thing.

Under the assumption that this is all under the existing preview feature, all these action names sound fine, and they're in-line with the existing ones.

Might be worth changing preview-realtime-stopped to emphasize that it's sort of an error-case? (preview-realtime-error-stopped?) Just from the action-name it sounds like something that's a user-action, and it's hard to say how it might be different from preview-realtime-off. (The more intuitive these are from the names without needing to check the docs the better.)

@DLynch no worries, thank you for the feedback. I updated to preview-realtime-error-stopped, it is more clear. Any other suggestions? Or do you think we can go ahead and work on the patch?

I think it sounds good to go ahead with.

In T306176#8092738, @DLynch wrote:

I think it sounds good to go ahead with.

I'll work on the patch and submit it for review. Thank you!

Change 816854 had a related patch set uploaded (by HMonroy; author: HMonroy):

[mediawiki/extensions/WikiEditor@master] Realtime Preview: add event tracking to triggered events

https://gerrit.wikimedia.org/r/816854

Change 816854 abandoned by HMonroy:

[mediawiki/extensions/WikiEditor@master] Realtime Preview: add event tracking to triggered events

Reason:

https://gerrit.wikimedia.org/r/816854

Change 816868 had a related patch set uploaded (by HMonroy; author: HMonroy):

[mediawiki/extensions/WikiEditor@master] Realtime Preview: add event tracking to triggered events

https://gerrit.wikimedia.org/r/816868

@DLynch I finally have a patch ready for review https://gerrit.wikimedia.org/r/816868. Thank you for your help and support on this ticket 😄

Change 816868 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@master] Realtime Preview: add event tracking to triggered events

https://gerrit.wikimedia.org/r/816868

@HMonroy just a few things I have observed so far.

preview-realtime-reload-error	Triggered when realtime preview pane shows an error message and the reload button is clicked	VisualEditorFeatureUse

When the error message is showing, the Reload button will only reload the preview if the wikitext has been changed. However, clicking the Reload button will always fire preview-realtime-reload-error, regardless of whether the preview is actually reloaded. A similar thing happens when you are in manual-mode (see the last sentence below).

It only fires the preview-realtime-reload-error if you click the "Reload" button in the error message. If you click the "Reload" button in the manual-mode bar or the shortcut it fires preview-realtime-reload-manual or preview-realtime-reload-hover.

preview-realtime-reload-hover	Triggered when the reload button that shows on hover in the realtime preview pane is clicked	VisualEditorFeatureUse

It also fires when you press the shortcut. I assume that is OK.

preview-realtime-reload-manual	Triggered when the reload button in the realtime preview pane manual bar is clicked	VisualEditorFeatureUse

It also fires when you press the shortcut. I assume that is OK.

When in manual-mode, the preview will only reload if the wikitext has been changed. However, the preview-realtime-reload-manual event always fires when clicking Reload or the shortcut, even if the preview is not actually reloaded.

@NRodriguez I have finished all the testing I can think of for now. I went through every transition between the different states of RTP (auto, manual, error) using the different methods (auto, button, shortcut). I have a couple of questions:

1. Of the 5 "Data Questions" above I think the data we are collecting will only help us answer the first two. Is this OK?
2. No event is fired when the preview reloads automatically. Is this OK?
3. Sometimes when you close then reopen RTP this causes the preview to reload, but there is no event for this. Is this OK?

Also see the couple of issues raised in T306176#8114615, namely:

• preview-realtime-reload-manual and preview-realtime-reload-error can fire even if the preview is not reloaded.
• In an error condition, preview-realtime-reload-manual or preview-realtime-reload-hover can be fired instead of preview-realtime-reload-error.

Test environment: https://en.wikipedia.beta.wmflabs.org WikiEditor 0.5.3 (267338d) 16:16, 29 July 2022.
Test browsers: Firefox 91, Chromium 87.

1. Of the 5 "Data Questions" above I think the data we are collecting will only help us answer the first two. Is this OK?

Yes, while not ideal, this is ok.

2. No event is fired when the preview reloads automatically. Is this OK?

Let's begin to track this event, that's the highest fidelity to people using the feature (more than clicking on the button in toolbar or manual refresh. @HMonroy, do you prefer to add it to the scope of this ticket or make a new one?

3. Sometimes when you close then reopen RTP this causes the preview to reload, but there is no event for this. Is this OK?

See answer above, if we start tracking the preview reload, this issue will no longer exsit.

Thanks for structuring the questions in the way that you did @dom_walden. @HMonroy, lmk what is needed! I assume we can use similar naming convention for the event as we have with other events!

cc @MNeisler

Let's begin to track this event, that's the highest fidelity to people using the feature (more than clicking on the button in toolbar or manual refresh. @HMonroy, do you prefer to add it to the scope of this ticket or make a new one?

My vote is adding it to the scope of this ticket so that we can easier track the work (code and data dictionary).

Proposed new event name	Action that will trigger new event	Schema where event will be logged
preview-realtime-loaded	Triggered when realtime preview requests and successfully returns a reload.	VisualEditorFeatureUse

@ppelberg @DLynch @MNeisler are you okay with adding the above event to the New instrumentation in this ticket?

In T306176#8114615, @dom_walden wrote:

@HMonroy just a few things I have observed so far.

preview-realtime-reload-error	Triggered when realtime preview pane shows an error message and the reload button is clicked	VisualEditorFeatureUse

When the error message is showing, the Reload button will only reload the preview if the wikitext has been changed. However, clicking the Reload button will always fire preview-realtime-reload-error, regardless of whether the preview is actually reloaded. A similar thing happens when you are in manual-mode (see the last sentence below).

It only fires the preview-realtime-reload-error if you click the "Reload" button in the error message. If you click the "Reload" button in the manual-mode bar or the shortcut it fires preview-realtime-reload-manual or preview-realtime-reload-hover.

@dom_walden Perhaps, Triggered when realtime preview pane shows an error message and the reload button is clicked needs to be more clear. preview-realtime-reload-error is triggered when the reload button under the error that says Preview is not loading is clicked. The event doesn't know whether the pane actually reloads, the event is triggered if the button was clicked. The same for manual mode, the event doesn't know whether the pane was actually refreshed but that the button was clicked.

This brings up a great point!
@NRodriguez do you want to trigger these events on the reload clicks or you only want to track these events when the click actually triggered the reload?

@NRodriguez do you want to trigger these events on the reload clicks or you only want to track these events when the click actually triggered the reload?

If we track the new event when the reload is actually triggered (agnostic of how it was summoned-- shortcut or click) then the click count should be just when people clicked it, and the reload preview-realtime-loaded will give us the actual reload count number

Hope that answers question!

Change 820242 had a related patch set uploaded (by HMonroy; author: HMonroy):

[mediawiki/extensions/WikiEditor@master] Realtime Preview: add event tracking to load action

https://gerrit.wikimedia.org/r/820242

@ppelberg @DLynch @MNeisler I was bold, and updated the New Instrumentation section. I added the preview-realtime-loaded. Please let me know if there's any objections with this addition. There's a patch ready for review with this change (https://gerrit.wikimedia.org/r/820242)

@dom_walden Please let me know if you notice any inconsistencies with the realtime preview event logging once it's been merged. Thank you for your feedback!

Change 820242 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@master] Realtime Preview: add event tracking to load action

https://gerrit.wikimedia.org/r/820242

In T306176#8129998, @HMonroy wrote:

@dom_walden Please let me know if you notice any inconsistencies with the realtime preview event logging once it's been merged. Thank you for your feedback!

Thanks.

preview-realtime-loaded	Triggered when realtime preview auto-loads the realtime preview pane.	VisualEditorFeatureUse

I've been testing on beta. preview-realtime-loaded fires for me every time the preview finishes reloading, regardless of the method (auto, shortcut, button, manual, etc.) or whether it reloads successfully or leads to an error. Is this ok? Do we need to update the "New Instrumentation" table to reflect this?

In T306176#8131267, @dom_walden wrote:

In T306176#8129998, @HMonroy wrote:

@dom_walden Please let me know if you notice any inconsistencies with the realtime preview event logging once it's been merged. Thank you for your feedback!

Thanks.

preview-realtime-loaded	Triggered when realtime preview auto-loads the realtime preview pane.	VisualEditorFeatureUse

I've been testing on beta. preview-realtime-loaded fires for me every time the preview finishes reloading, regardless of the method (auto, shortcut, button, manual, etc.) or whether it reloads successfully or leads to an error. Is this ok? Do we need to update the "New Instrumentation" table to reflect this?

I think that is good. I can update the New Instrumentation.

@NRodriguez I just want to make sure this is the case. Event preview-realtime-loaded is being triggered for every case the realtime preview is loaded also when there is an error. Please let us know if this is okay; otherwise, I can change it so that this event is only triggered on successful reloading.

Please let us know if this is okay; otherwise, I can change it so that this event is only triggered on successful reloading.

We discussed in RTL, since the error event is being tracked independently, this is ok! Thanks all

In T306176#8138398, @NRodriguez wrote:

Please let us know if this is okay; otherwise, I can change it so that this event is only triggered on successful reloading.

We discussed in RTL, since the error event is being tracked independently, this is ok! Thanks all

Do we track the error event? We have an event when you click the reload button in the error message, is that what we mean?

Unless the client-side error gets sent to logstash (@Samwilson might know?)

Otherwise, I don't have anymore testing to do here.

Do we track the error event? We have an event when you click the reload button in the error message, is that what we mean?

Yup, we will know if they click the reload button in the error message so this will let us know if there was an error when they click on the link, but we will not know if there was an error if the user does not click on the link in the error message.

When preview loading errors, the client runs mw.log.error( 'WikiEditor realtime preview error', result );, but it's not an error event that is being tracked.

@HMonroy did you complete this step?

Update schema data dictionary with the events spec'd in this task once they have been verified to be implemented as expected. Read: clients are emitting events as expected and said events are being logged in the database as expected as well.

Down to cut a separate ticket for this so I can resolve this one if not!

In T306176#8163021, @NRodriguez wrote:

@HMonroy did you complete this step?

Update schema data dictionary with the events spec'd in this task once they have been verified to be implemented as expected. Read: clients are emitting events as expected and said events are being logged in the database as expected as well.

Down to cut a separate ticket for this so I can resolve this one if not!

I've cut a new ticket to address updating data dictionary with all the realtime preview events that were added in this feature.