it looks like we can go ahead and add logging for the existing live-preview button

@Samwilson @NRodriguez: Hi! :) Sorry to barge in but I see this discussion is progressing quickly (which is good!) and I just wanted to make sure you've chatted with someone from Product Analytics (as stakeholders of EAS data and owners of EAS schema) before proceeding with actual modifications.

I wanted to let you know that I chatted with @KSiebert about this and I proposed booking office hours with @MNeisler for consultation on this work. Megan (on vacation until Friday, October 8) is the Product Analytics point person for Editing & Language, has extensive experience with instrumentation design & especially EditAttemptStep, and would be able to offer guidance around specifics of the "preview" events (such as their frequency and whether any additional data should be collected), and would be able to code review EAS patch for this.

Lmk if this helps clarify the future of the project and how it impacts the current scope of this ticket

@mpopov sounds great, looking forward to talking to megan

and @Samwilson lmk if in the meantime this new annotated diagram changes any of your outlined approach

@mpopov: No worries, I was just setting out the plan to make sure it sounds okay. Not going to start work on it yet. :-) I think we'll create separate tickets for the various bits of it.

@NRodriguez that diagram looks correct! The only thing that I'd clarify is that "Full page preview" can also mean non-live (i.e. the whole page reloads). So far we're not planning on tracking that with EditAttemptStep, although probably that could be talked about more because it might be nice to end up with all the data in one place?

@NRodriguez that diagram looks correct! The only thing that I'd clarify is that "Full page preview" can also mean non-live (i.e. the whole page reloads). So far we're not planning on tracking that with EditAttemptStep, although probably that could be talked about more because it might be nice to end up with all the data in one place?

yeah i think it'd be great to track full page preview in one place as well! excited to hear what the best schema for this data is

Notes from me and @KSiebert talking today:

• action = preview
• preview_type =
  • realtime turned on (wikieditor)
  • realtime already-on when page loads (wikieditor)
  • realtime turned off (wikieditor)
  • live (core, JS)
  • fullpage (core, no JS)

After discussion earlier today, we've decided to use the visualeditorfeatureuse schema, with events having feature: "preview" and the following action values (to fit in with the existing data dictionary):

• preview-nonlive – The main 'Preview' button was clicked to trigger a non-live preview (i.e. full-page reload).
• preview-live – The main 'Preview' button was clicked to trigger a live preview.
• preview-realtime-on – The WikiEditor 'Preview' button was clicked to turn the preview panel on.
• preview-realtime-off – The WikiEditor 'Preview' button was clicked to turn the preview panel off.
• preview-realtime-inuse – Fired without user interaction when the WikiEditor preview feature is already on when the editing form is opened.

I'll update the task descriptions with these.

@ppelberg I've updated the above, but I'm not sure what the "Event name" field is. Is it the same as the feature name?

@Samwilson Thanks for updating with the proposed changes, I've gone ahead and updated the table on the task description-- let us know if the event name being blank is an issue. Pinging the folks that are tagged as owners of the next step on the handy table! @DLynch @MNeisler
Thanks everyone 😄

In T290521#7373549, @Samwilson wrote:

Can we use the druid.event_navigationtiming table to look at ?action=submit?

That should include (non-AJAX) previews but also diffs and unsuccessful save attempts (edit conflict etc).

In T290521#7430428, @Samwilson wrote:

After discussion earlier today, we've decided to use the visualeditorfeatureuse schema, with events having feature: "preview" and the following action values (to fit in with the existing data dictionary):

• preview-nonlive – The main 'Preview' button was clicked to trigger a non-live preview (i.e. full-page reload).
• preview-live – The main 'Preview' button was clicked to trigger a live preview.
• preview-realtime-on – The WikiEditor 'Preview' button was clicked to turn the preview panel on.
• preview-realtime-off – The WikiEditor 'Preview' button was clicked to turn the preview panel off.
• preview-realtime-inuse – Fired without user interaction when the WikiEditor preview feature is already on when the editing form is opened.

@Samwilson Sorry for the delay! I've reviewed and confirmed with @DLynch that the proposed feature and action value names look good for inclusion as part of VisualEditorFeatureUse. No changes needed so you can move forward with implementation.

Change 735469 had a related patch set uploaded (by Samwilson; author: Samwilson):

[mediawiki/extensions/WikimediaEvents@master] Log a 'preview-nonlive' action when a page is previewed

https://gerrit.wikimedia.org/r/735469

Change 735570 had a related patch set uploaded (by Samwilson; author: Samwilson):

[mediawiki/extensions/WikimediaEvents@master] Log live-preview usage

https://gerrit.wikimedia.org/r/735570

The above patch r735570 adds logging whenever someone clicks the 'preview' button, and not only when they view a preview (which generally should be the same, but there could be issues that prevent the viewing). Is this how we want to do it? It seems to capture the user interaction best, but could also result in false logging (e.g. someone double-clicking the button).

The other patch above (r735469) adds server-side logging for non-live previews.

Change 735570 abandoned by Samwilson:

[mediawiki/extensions/WikimediaEvents@master] Log live-preview usage

Reason:

Will add the logging to WikiEditor instead of here.

https://gerrit.wikimedia.org/r/735570

Change 736324 had a related patch set uploaded (by Samwilson; author: Samwilson):

[mediawiki/extensions/WikiEditor@master] Add live-preview logging

https://gerrit.wikimedia.org/r/736324

Change 736645 had a related patch set uploaded (by Samwilson; author: Samwilson):

[mediawiki/extensions/WikiEditor@master] Add logging for non-live previews

https://gerrit.wikimedia.org/r/736645

Change 736646 had a related patch set uploaded (by Samwilson; author: Samwilson):

[mediawiki/extensions/WikiEditor@master] Avoid User::getEditCount() and ::getOption()

https://gerrit.wikimedia.org/r/736646

Change 735469 abandoned by Samwilson:

[mediawiki/extensions/WikimediaEvents@master] Log a 'preview-nonlive' action when a page is previewed

Reason:

Will log in WikiEditor instead, in order to keep all preview logging together in one codebase.

https://gerrit.wikimedia.org/r/735469

Change 736646 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@master] Avoid User::getEditCount() and ::getOption()

https://gerrit.wikimedia.org/r/736646

Hello @DLynch and @ppelberg, Sam has implemented changes so I've gone ahead and updated the table in this task. The next step is for you to review those changes and let us know if there are any notes! We have a review column in our kanban, but we figured it may make more sense in yours. Let us know how we can support and thanks for all of your help.

Just to note here: the current reviews are for existing preview features only, and we'll have to come back and go through this process again when we've built the realtime-preview feature, to add the remaining three log actions.

In T290521#7510391, @NRodriguez wrote:

Hello @DLynch and @ppelberg, Sam has implemented changes so I've gone ahead and updated the table in this task. The next step is for you to review those changes and let us know if there are any notes! We have a review column in our kanban, but we figured it may make more sense in yours. Let us know how we can support and thanks for all of your help.

Noted! I'm pulling this task onto our current workboard to signal this is ready for our review.

Thank you @ppelberg

Change 736645 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@master] Add logging for non-live previews

https://gerrit.wikimedia.org/r/736645

Change 736324 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@master] Add live-preview logging

https://gerrit.wikimedia.org/r/736324

@MNeisler the changes are going live on the train this week so I went ahead and updated the table! Next step on deck lists out Product Analytics as owner:

Verify newly instrumented events are being logged in database as expected, once the new instrumentation has landed in production.

Thanks for all your help 🥳

Followup ticket to come for when we implement real-time preview

I've added the documentation of the new feature and first two actions.

Follow-up for logging the remaining three actions is T298218: Add EventLogging for realtime preview.

I don't think there's anything left to do for this task.

For QA: check that events are fired at the right times and no others, and data is received and query-able in Superset.

I don't think there's anything left to do for this task.

Actually, one last point in the table above for Analytics to tick off: Verify newly instrumented events are being logged in database as expected, once the new instrumentation has landed in production.

@MNeisler Let us know if all looks good on your end and we can resolve! Thanks

@NRodriguez

Thanks for the ping! I reviewed the aggregate data logged in visualeditorfeatureuse and confirmed that the newly instrumented events are being logged expected. Feel free to resolve.

Here are the results of a queck query checking total number of events logged for each of the two new preview actions.

Data via:

SELECT 
event.action,
event.platform,
event.editor_interface,
COUNT(*) AS num_events
FROM event.visualeditorfeatureuse 
WHERE 
(year = 2020 AND month = 12)
OR year = 2021
AND event.feature = 'preview'
GROUP BY 
event.action,
event.platform,
event.editor_interface