I've started to draft an instrumentation spec. This covers both existing and new instrumentation needed to track details on published edits using edit tags stored in mediawiki_history and user engagement with the proposed Edit Check mobile workflow to be tracked in EditAttemptStep and VisualEditorFeatureUse.

Note: This instrumentation spec is still in progress and will be reviewed and completed based on the measurement plan to be finalized in T325838.

Update: I worked with @nayoub to map the events proposed in the Edit Check instrumentation spec to the proposed UX mobile workflow. In the annotated workflow, the events in the green boxes are already instrumented and events in the yellow boxes are proposed new events to the instrument.

OPEN QUESTIONS (documented in the annotated workflow as well):

• Edit summary ("Share why you didn't add a citation" step of new workflow) instrumentation:

One of our metrics identified in the measurement plan is the proportion of published edits that include an explicit acknowledgment of why a citation was not added. In the proposed mobile workflow, there are currently two locations a user can provide a reason for not adding a citation: (1) by selecting one of the three provided reasons shown as part of the "dismiss adding a citation" step and 2) while typing the edit summary where they will receive a prompt that says: "Describe what you changed and why a citation might be needed". See T329593 for context

We are currently proposing to add events to track option 1 but option 2 may be more difficult to instrument. We might be able to instrument whether someone added a summary but parsing the text to understand if they provided a reason is much more complicated. Are we ok with just instrumenting option 1 (with the understanding we won't know if a reason was provided in the edit summary)? Is there any way to instrument usage of the edit summary that might help us understand if the user provided a reason?

• Ready to publish instrumentation:

In the mobile workflow ("Citation Added" step), after a user selects the ">" indicating they are ready to publish after adding a citation. Is that currently instrumented as event.action = 'saveIntent' or does this need to be instrumented?

Next Steps:

• Create annotated Desktop workflow if needed
• Share with Edit Team for review
• Finalize and define new instrumentation to be added

@nayoub

In the "dismiss adding a citation" step of the Edit Check workflow, will there be an explicit option to dismiss adding a citation because the user believed it appeared in error? For instance, the user was adding an image or just rearranging content on the page. Would both of these false positives examples be covered under "I didn't add new information" shown in this UX mobile mockup?

Context: I'm thinking through instrumentation requirements for this feature and want to check that we have a way to accurately calculate the rate of false positives (Edit Check appearing when it shouldn't). This will be useful in identifying some of the prioritized failure scenarios identified in the pre-mortem and assessing the accuracy of the feature.

In the mobile workflow ("Citation Added" step), after a user selects the ">" indicating they are ready to publish after adding a citation. Is that currently instrumented as event.action = 'saveIntent' or does this need to be instrumented?

Right now saveIntent will fire just before the actual save dialog opens, once the user is done with all their edit checks. (I'm not sure if this is semantically correct, insofar as the user has expressed their intent-to-save when they click the toolbar button and doesn't really care whether they're going to be getting the edit check or going immediately to the save dialog... but it does preserve the existing meaning of the timings involved.)

In T324735#8817159, @MNeisler wrote:

OPEN QUESTIONS (documented in the annotated workflow as well):

• Edit summary ("Share why you didn't add a citation" step of new workflow) instrumentation:

Is there any way to instrument usage of the edit summary that might help us understand if the user provided a reason?

@DLynch: can you think of an approach that would enable us to answer a question like, "What percentage of people who were shown an Edit Check interacted with (or ideally, published an edit summary) along with the edit they've made?

...my first instinct was to review mw:VisualEditor/FeatureUse data dictionary which doesn't seem to include any mention of edit summary instrumentation outside of DiscussionTools.

In T324735#8847900, @MNeisler wrote:

In the "dismiss adding a citation" step of the Edit Check workflow, will there be an explicit option to dismiss adding a citation because the user believed it appeared in error? For instance, the user was adding an image or just rearranging content on the page. Would both of these false positives examples be covered under "I didn't add new information" shown in this UX mobile mockup?

Great spot, @MNeisler and +1: I agree with you in thinking we should present people with an option that enables them to express, as you described, that they "...believed it appeared in error."

Let's use T329593 to define the language for this option and finalize the language for the other options we'll present people with in this moment.

@DLynch: can you think of an approach that would enable us to answer a question like, "What percentage of people who were shown an Edit Check interacted with (or ideally, published an edit summary) along with the edit they've made?

I don't think there's anything in the schemas currently for interacted-with. That said, it sounds like something that could be extracted from the database via a query -- though you might need to get fancy with it if you wanted to filter out auto-generated summaries (the ones like /* section heading */).

@DLynch
I've defined a list of new events to be instrumented to track engagement with the Edit Check feature. These are highlighted in yellow in the instrumentation spec along with suggested event values to log in the visualeditorfeatureuse schema. Can you please review and let me know if you have questions or other suggestions on how to best instrument these?

A few notes:

• I took an initial pass at mapping these events to existing VEFU actions where applicable but noted where I think new ones should be added (See the Notes/Discussion column for my rationale). I'm flexible on the values selected though as long as I can accurately distinguish these events in the data.
• I've also added these events to the desktop and mobile UX workflows on this miro board if it would help to see them mapped to the actual workflow.
• I'm recommending these events be tracked under the proposed feature value feature = 'editCheckReferences'. I'm open to other feature name suggestions as well but think it would be good to (1) align with other naming conventions used for this feature elsewhere (.e.g the current tag (#editcheck-references)) and (2) be specific enough so we can distinguish this feature from other edit check types in the future.
• I've confirmed that I should be able to extract needed data on edit summaries from the database via query so no new instrumentation is needed for that component.

Open Question:
We'd like to instrument each option presented to the user in the "dismiss adding a citation" dialog (i.e. action ='dialog-choose-[name of option selected]') but the exact options are still being discussed and finalized in T329593. Do these options need to be finalized before this is instrumented or is there some way that the option the user selects can be automatically passed to the event value based on what is presented to the user?

@MNeisler The wording doesn't need to be finalized -- as currently implemented (here) there's some simple keys associated with the options (no-info, already-cited, not-sure, other), and we can just use those in the action name.

Re: feature names, we're going to get some automated logging that'll be under editCheckInspector and editCheckDialog because that's the level the automated systems work at. editCheckReferences can be used for anything explicitly logged for this check, of course.

Change 948616 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@master] Instrumentation for edit check features

https://gerrit.wikimedia.org/r/948616

I've updated the description to include the new events that are triggered.

In T324735#9120588, @DLynch wrote:

I've updated the description to include the new events that are triggered.

Wonderful – thank you, @DLynch.

@Ryasmeen, does anything about the description's Instrumentation Requirements section seem unclear/missing to you?

Change 948616 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Instrumentation for edit check features

https://gerrit.wikimedia.org/r/948616

I checked all the events on both Mobile and Desktop. Here are few of my observations:

1. Double logging of VEFU feature: editCheckReferences action: context-show:
   
   In both desktop and mobile: the first event is being emitted twice when the edit check references dialog gets triggered. This happens for the following cases when this dialog is presented

• When you enter enough text to trigger the check and press "save".
• When you click on 'x' on "Add a Citation".
  
  However, it does not get emitted twice when you click on the "<" on the rejection reason window.

2. There is an event that is being recorded that is missing on the table mentioned on the description:
   
   visualEditorFeatureUse feature: 'editCheckReferencesInspector', action: 'dialog-abort'
   
   This event is being emitted when you click on the "<" on the rejection reason window.

3. I also noticed that the event mapping on the Miro board maybe needs some updating. There is a mapping of the following event to the UX workflow where an editor closes the Add a citation dialog by clicking outside the dialog which is not a valid scenario since we cannot close the citation dialog.
   
   visualEditorFeatureUse feature: editCheckReferences action: context-show

I think 2 is fine, it's just a close event for a phase of the inspector (and isn't a doubling-up with any other way you can "abort" that particular dialog). It's okay to have extra events mixed in, given the automatic nature of many of these, so long as the ones the spreadsheet describes aren't appearing in ways they shouldn't.

1... I actually fixed almost a month ago but we've not updated the submodule since then. I'll make that happen.

Change 962711 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@master] Update VE core submodule to master (ca5f6c26d)

https://gerrit.wikimedia.org/r/962711

Change 962711 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Update VE core submodule to master (ca5f6c26d)

https://gerrit.wikimedia.org/r/962711

@DLynch: For the double-logging issue, I see it got fixed for the first scenario. But for the second scenario, where the user clicks on "Yes" on the "Add a citation", and then clicks on 'x' on the same dialog, it is still triggering the event twice.

Change #1038440 had a related patch set uploaded (by DLynch; author: DLynch):

[VisualEditor/VisualEditor@master] LinearContext: make context items aware when they're being refreshed

https://gerrit.wikimedia.org/r/1038440