Page MenuHomePhabricator

Real-time Preview: Data Questions
Open, Needs TriagePublic5 Estimated Story Points

Description

Inside the project update of the Real Time Preview wish, the document posed some open data questions that would inform the direction of the wish and help us measure its success once we release the changes.

The scope of this ticket can be limited to:

  • Implementing the tracking if it does NOT exist
  • Creating the place where a person can pull that data from a dashboard if/once it DOES exist

For the following data question which will help deepen our understanding of the problem:

How many 2010 wikitext editors preview their changes?
    Does previewing changes lead to less reverts?

New instrumentation

The list below should contain an exhaustive list of the new events y'all will be implementing.

  • "Action" = the action that will cause an event to be emitted.
  • "Event name" = the proposed event name
  • "Schema" = VisualEditorFeatureUse β€” the schema(s) within which said event will be logged

The event will be of the form {feature: "preview", action: x} where x is one of:

  • preview-nonlive – The main 'Preview' button was clicked to trigger a non-live preview (i.e. full-page reload).
  • preview-live – The main 'Preview' button was clicked to trigger a live preview.
  • preview-realtime-on – The WikiEditor 'Preview' button was clicked to turn the preview panel on.
  • preview-realtime-off – The WikiEditor 'Preview' button was clicked to turn the preview panel off.
  • preview-realtime-inuse – Fired without user interaction when the WikiEditor preview feature is already on when the editing form is opened.

Process

StepPerson(s) responsibleDescriptionStatus
1.Community TechPopulate the ===New instrumentation table with proposed changes, once defined done
2.@MNeisler + @DLynchReview the "Event names" Community Tech is proposing to introduce done
3.Community TechImplement the instrumentation changes defined above done
4.Editing TeamCode review the instrumentation changes Community Tech will have implemented done
5.Community TechVerify newly instrumented events are being emitted as expected by clients in progress
6.Product AnalyticsVerify newly instrumented events are being logged in database as expected, once the new instrumentation has landed in production.
7.Community TechUpdate schema documentation with the events spec'd in this task once they have been verified to be implemented as expected. Read: clients are emitting events as expected and said events are being logged in the database as expected as well.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Inside of the editing channel, a collaborator shared some knowledge on possible avenues for tracking this

Instrumentation
what percentage of editors preview their edits before publishing?
to answer this question, for both the wikieditor, new wikitext editor, and visualeditor, you > would be able to leverage the mwSave.review-initial-visual and mwSave.review-initial-source events. note: i took "preview their edits" above to mean "the percentage of editors who clicked "Review your changes" in the save dialog. [i]

ldelench_wmf set the point value for this task to 2.Sep 9 2021, 2:41 PM
NRodriguez renamed this task from Real-time Preview: Data Investigations to Real-time Preview: Data Questions.Sep 9 2021, 5:06 PM
NRodriguez removed the point value for this task.
ldelench_wmf set the point value for this task to 5.Sep 9 2021, 5:15 PM

I'm not sure where the mwSave.review-initial-visual and mwSave.review-initial-source events can be queried, but they both are only applicable to VE (as far as I know). We're only concerned with the 2003 and 2010 editors with this project.

Can we use the druid.event_navigationtiming table to look at ?action=submit? Here's a Superset chart of that. The numbers seem pretty small, but they're not counting the live-preview previews (which are done via the API, action=parse), so maybe are correct?

daily-counts-of-action-submit-2021-09-23T06-06-01.927Z.jpg (698Γ—1 px, 142 KB)

For future tracking of previews (assuming we're adding the new functionality to the WikiEditor extension, and not core) it sounds like the EditAttemptStep logging schema would be a suitable place, although adding a new preview step to that probably requires some discussion. We might want to add a preview_type as well (to distinguish between live-preview and real-time preview).

(Overall, event logging is not an area I'm very familiar with so maybe I'm on quite the wrong track here!)

Hello @Samwilson

We're only concerned with the 2003 and 2010 editors with this project.

Correct! Unsure what is best practice here in terms of our databases and schemas

it sounds like the EditAttemptStep logging schema would be a suitable place, although adding a new preview step to that probably requires some discussion. We might want to add a preview_type as well (to distinguish between live-preview and real-time preview).

One thing I'd flag is it'd be great to filter by type of editing interface

We might want to add a preview_type as well (to distinguish between live-preview and real-time preview).

Right now I think as long as we track the preview type being "Full page preview" and "In-editor preview" it will be enough for our purposes.

But I'd like to understand the benchmark before we begin work. So right now, the goal is simply to track--- who is clicking on "show preview" in our editing flows? Not sure what the right schema is for that so maybe someone in data science department can chime in?

Right now I think as long as we track the preview type being "Full page preview" and "In-editor preview" it will be enough for our purposes.

Here, are you calling the live-preview "Full page preview"? i.e. a different thing from the non-live full-page preview that requires a full (browser) page refresh? (The latter is the ?action=submit type.)

Anyway, yeah it looks like we can go ahead and add logging for the existing live-preview button. I think this will require:

  1. Altering the EditAttemptStep schema to add a new action value (of previewLive);
  2. Adding a hook to the mediawiki.action.edit.preview module in core;
  3. Using this hook in MediaWiki-extensions-WikimediaEvents, to add the actual logging code;
  4. Creating a new Superset chart to show the data.

Later, when we have built the realtime preview in WikiEditor, we can follow a similar process to add another action value of previewRealtime.

it looks like we can go ahead and add logging for the existing live-preview button

@Samwilson @NRodriguez: Hi! :) Sorry to barge in but I see this discussion is progressing quickly (which is good!) and I just wanted to make sure you've chatted with someone from Product Analytics (as stakeholders of EAS data and owners of EAS schema) before proceeding with actual modifications.

I wanted to let you know that I chatted with @KSiebert about this and I proposed booking office hours with @MNeisler for consultation on this work. Megan (on vacation until Friday, October 8) is the Product Analytics point person for Editing & Language, has extensive experience with instrumentation design & especially EditAttemptStep, and would be able to offer guidance around specifics of the "preview" events (such as their frequency and whether any additional data should be collected), and would be able to code review EAS patch for this.

tracking spec.png (980Γ—1 px, 703 KB)

Lmk if this helps clarify the future of the project and how it impacts the current scope of this ticket

@mpopov sounds great, looking forward to talking to megan

and @Samwilson lmk if in the meantime this new annotated diagram changes any of your outlined approach

@mpopov: No worries, I was just setting out the plan to make sure it sounds okay. Not going to start work on it yet. :-) I think we'll create separate tickets for the various bits of it.

@NRodriguez that diagram looks correct! The only thing that I'd clarify is that "Full page preview" can also mean non-live (i.e. the whole page reloads). So far we're not planning on tracking that with EditAttemptStep, although probably that could be talked about more because it might be nice to end up with all the data in one place?

@NRodriguez that diagram looks correct! The only thing that I'd clarify is that "Full page preview" can also mean non-live (i.e. the whole page reloads). So far we're not planning on tracking that with EditAttemptStep, although probably that could be talked about more because it might be nice to end up with all the data in one place?

yeah i think it'd be great to track full page preview in one place as well! excited to hear what the best schema for this data is

Notes from me and @KSiebert talking today:

  • action = preview
  • preview_type =
    • realtime turned on (wikieditor)
    • realtime already-on when page loads (wikieditor)
    • realtime turned off (wikieditor)
    • live (core, JS)
    • fullpage (core, no JS)
ppelberg updated the task description. (Show Details)
ppelberg added a subscriber: DLynch.

After discussion earlier today, we've decided to use the visualeditorfeatureuse schema, with events having feature: "preview" and the following action values (to fit in with the existing data dictionary):

  • preview-nonlive – The main 'Preview' button was clicked to trigger a non-live preview (i.e. full-page reload).
  • preview-live – The main 'Preview' button was clicked to trigger a live preview.
  • preview-realtime-on – The WikiEditor 'Preview' button was clicked to turn the preview panel on.
  • preview-realtime-off – The WikiEditor 'Preview' button was clicked to turn the preview panel off.
  • preview-realtime-inuse – Fired without user interaction when the WikiEditor preview feature is already on when the editing form is opened.

I'll update the task descriptions with these.

Samwilson added a subscriber: ppelberg.

@ppelberg I've updated the above, but I'm not sure what the "Event name" field is. Is it the same as the feature name?

@Samwilson Thanks for updating with the proposed changes, I've gone ahead and updated the table on the task description-- let us know if the event name being blank is an issue. Pinging the folks that are tagged as owners of the next step on the handy table! @DLynch @MNeisler
Thanks everyone πŸ˜„

Can we use the druid.event_navigationtiming table to look at ?action=submit?

That should include (non-AJAX) previews but also diffs and unsuccessful save attempts (edit conflict etc).

After discussion earlier today, we've decided to use the visualeditorfeatureuse schema, with events having feature: "preview" and the following action values (to fit in with the existing data dictionary):

  • preview-nonlive – The main 'Preview' button was clicked to trigger a non-live preview (i.e. full-page reload).
  • preview-live – The main 'Preview' button was clicked to trigger a live preview.
  • preview-realtime-on – The WikiEditor 'Preview' button was clicked to turn the preview panel on.
  • preview-realtime-off – The WikiEditor 'Preview' button was clicked to turn the preview panel off.
  • preview-realtime-inuse – Fired without user interaction when the WikiEditor preview feature is already on when the editing form is opened.

@Samwilson Sorry for the delay! I've reviewed and confirmed with @DLynch that the proposed feature and action value names look good for inclusion as part of VisualEditorFeatureUse. No changes needed so you can move forward with implementation.

Change 735469 had a related patch set uploaded (by Samwilson; author: Samwilson):

[mediawiki/extensions/WikimediaEvents@master] Log a 'preview-nonlive' action when a page is previewed

https://gerrit.wikimedia.org/r/735469

Change 735570 had a related patch set uploaded (by Samwilson; author: Samwilson):

[mediawiki/extensions/WikimediaEvents@master] Log live-preview usage

https://gerrit.wikimedia.org/r/735570

The above patch r735570 adds logging whenever someone clicks the 'preview' button, and not only when they view a preview (which generally should be the same, but there could be issues that prevent the viewing). Is this how we want to do it? It seems to capture the user interaction best, but could also result in false logging (e.g. someone double-clicking the button).

The other patch above (r735469) adds server-side logging for non-live previews.

Change 735570 abandoned by Samwilson:

[mediawiki/extensions/WikimediaEvents@master] Log live-preview usage

Reason:

Will add the logging to WikiEditor instead of here.

https://gerrit.wikimedia.org/r/735570

Change 736324 had a related patch set uploaded (by Samwilson; author: Samwilson):

[mediawiki/extensions/WikiEditor@master] Add live-preview logging

https://gerrit.wikimedia.org/r/736324

Change 736645 had a related patch set uploaded (by Samwilson; author: Samwilson):

[mediawiki/extensions/WikiEditor@master] Add logging for non-live previews

https://gerrit.wikimedia.org/r/736645

Change 736646 had a related patch set uploaded (by Samwilson; author: Samwilson):

[mediawiki/extensions/WikiEditor@master] Avoid User::getEditCount() and ::getOption()

https://gerrit.wikimedia.org/r/736646

Change 735469 abandoned by Samwilson:

[mediawiki/extensions/WikimediaEvents@master] Log a 'preview-nonlive' action when a page is previewed

Reason:

Will log in WikiEditor instead, in order to keep all preview logging together in one codebase.

https://gerrit.wikimedia.org/r/735469

Change 736646 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@master] Avoid User::getEditCount() and ::getOption()

https://gerrit.wikimedia.org/r/736646

Hello @DLynch and @ppelberg, Sam has implemented changes so I've gone ahead and updated the table in this task. The next step is for you to review those changes and let us know if there are any notes! We have a review column in our kanban, but we figured it may make more sense in yours. Let us know how we can support and thanks for all of your help.

Just to note here: the current reviews are for existing preview features only, and we'll have to come back and go through this process again when we've built the realtime-preview feature, to add the remaining three log actions.

Hello @DLynch and @ppelberg, Sam has implemented changes so I've gone ahead and updated the table in this task. The next step is for you to review those changes and let us know if there are any notes! We have a review column in our kanban, but we figured it may make more sense in yours. Let us know how we can support and thanks for all of your help.

Noted! I'm pulling this task onto our current workboard to signal this is ready for our review.

I'm passing this back so it can move on to step 5.

Change 736645 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@master] Add logging for non-live previews

https://gerrit.wikimedia.org/r/736645

Change 736324 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@master] Add live-preview logging

https://gerrit.wikimedia.org/r/736324