Instrument each step of mobile VE edit flow
Closed, ResolvedPublic
Actions

Description

We intend for the improvements we are making to the mobile VE editing toolbar and the steps within the edit flow [1] to help a greater number of people complete their edits.

In order for us to be able to evaluate how effective our efforts have been, we'll need to be able to answer: How do people "move through" the edit flow?

And for us to be able to answer this question, we'll need instrumentation to track the actual actions/steps contributors are taking within the "edit flow".

What we know

Currently, we know:

When the editing interface starts to load: init
When the editing interface is ready to be interacted with: ready
When a contributor advances to the "Save your changes" screen/dialog: saveIntent
When a contributor tries to save an edit: saveAttempt
When a contributor successfully saves an edit: saveSuccess
When a contributor unsuccessfully ties to save an edit: saveFailure
When a contributor aborts their edit: abort
- The conditions under which a contributor aborts. Examples:
  - No changes have been made: nochange
  - Editor has be initialised but is not yet ready: preinit
  - User is switching from one editor to another, retaining changes: switchwith
  - User is switching from one editor to another, discarding changes: switchwithout
  - User is switching from one editor to another, and the content block is unchanged / transaction stack is empty: switchnochange
  - User is exiting the editor, discarding changes: abandon
  - User is exiting the editor, discarding changes, after a saveAttempt has been initiated but before saveSuccess/saveFailure.": abandonMidsave

Source: Schema:EditAttemptStep

We would like to know ("Done")

After this work is complete, we would like to know:

When a contributor "starts" an edit
- "Starts" could mean when a contributor makes any kind of change to the article
- Read: when the first transaction is made
When a contributors taps “<” in “Save your changes” dialog
When a contributors taps “<” in “Review your changes” view
When a contributor taps to "Preview"/"Review" their changes
When a contributor taps "Visual" or "Wikitext" in “Review your changes” view
When a contributor taps the "This is a minor edit" checkbox in the Save dialog
When a contributor taps the "Watch this page" checkbox in the Save dialog
When a contributor taps the “>” in v1 of the toolbar

This work also relates to our investigation into mobile VE's loading performance: T227930.

T211255: [EPIC] Mobile VE toolbar improvements

Details

Related Gerrit Patches:

mediawiki/extensions/VisualEditor : master	MWSaveDialog: add more instrumentation
mediawiki/extensions/WikiEditor : master	Implement firstChange logging
mediawiki/extensions/MobileFrontend : master	Add new EditAttemptStep stage: firstChange
mediawiki/extensions/VisualEditor : master	Add new EditAttemptStep stage: firstChange
mediawiki/extensions/WikimediaEvents : master	Update EditAttemptStep version
mediawiki/extensions/WikimediaEvents : master	Update EditAttemptStep

Related Objects
Search...

		Status	Assigned	Task
		Resolved	ppelberg	T229079 Instrument each step of mobile VE edit flow
		Open	Ryasmeen	T232790 Verify editing flow events are firing properly

Event Timeline

ppelberg created this task.Jul 26 2019, 12:45 AM

Restricted Application added a project: VisualEditor. · View Herald TranscriptJul 26 2019, 12:45 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

ppelberg renamed this task from Instrument each step of the edit flow to Instrument each step of mobile VE edit flow.Jul 26 2019, 12:49 AM

ppelberg updated the task description. (Show Details)Jul 26 2019, 6:07 PM

ppelberg updated the task description. (Show Details)Jul 26 2019, 6:23 PM

ppelberg updated the task description. (Show Details)Jul 30 2019, 7:52 PM

ppelberg mentioned this in T221316: Create measurement specifications and experiment plan.Jul 31 2019, 2:06 PM

ppelberg updated the task description. (Show Details)Aug 1 2019, 11:51 PM

Moving this to our current workboard for @DLynch or @Esanders to pick up

JTannerWMF moved this task from Incoming to Ready for Development on the VisualEditor (Current work) board.Aug 21 2019, 11:54 AM

Updates to task description to include the following events in the ...we would like to know section:

Tapping “<” in “Save your changes” dialog
Tapping “<” in “Review your changes” view
Tapping “Visual” or "Wikitext" in "Review your changes" view
Tapping “>” in the v1 toolbar

ppelberg updated the task description. (Show Details)Aug 29 2019, 3:17 PM

ppelberg updated the task description. (Show Details)Aug 29 2019, 6:17 PM

Updates to task description:

Removing T229030 as a dependency
Removing "⚠️This is a tracker task... message
Removing "This is likely to change considering we are planning to show the visual diff screen view by default. See the description of: T229030"

Documenting notes/decisions from a conversation today with @MNeisler and @Neil_P._Quinn_WMF:

The following events should should live in Schema:VisualEditorFeaureUse:
- "When a contributor "starts" an edit"
The following events should should live in Schema:EditAttemptStep
- When a contributors taps “<” in “Save your changes” dialog
- When a contributors taps “<” in “Review your changes” view
- When a contributor taps to "Preview"/"Review" their changes
- When a contributor taps "Visual" or "Wikitext" in “Review your changes” view
- When a contributor taps the "This is a minor edit" checkbox in the Save dialog
- When a contributor taps the "Watch this page" checkbox in the Save dialog
- When a contributor taps the “>” in v1 of the toolbar

@DLynch, are there any other questions you are needing answers to before work on adding instrumentation for the events listed in the NAME above can start?

In T229079#5466100, @ppelberg wrote:

Documenting notes/decisions from a conversation today with @MNeisler and @Neil_P._Quinn_WMF:

The following events should should live in Schema:VisualEditorFeaureUse:

"When a contributor "starts" an edit"

The following events should should live in Schema:EditAttemptStep

When a contributors taps “<” in “Save your changes” dialog

When a contributors taps “<” in “Review your changes” view

When a contributor taps to "Preview"/"Review" their changes

When a contributor taps "Visual" or "Wikitext" in “Review your changes” view

When a contributor taps the "This is a minor edit" checkbox in the Save dialog

When a contributor taps the "Watch this page" checkbox in the Save dialog

When a contributor taps the “>” in v1 of the toolbar

That's actually backwards; we thought starting an edit should be in EditAttemptStep and the rest should be in VisualEditorFeatureUse 😁

In T229079#5467820, @Neil_P._Quinn_WMF wrote:

That's actually backwards; we thought starting an edit should be in EditAttemptStep and the rest should be in VisualEditorFeatureUse 😁

Shoot, yes. Good catch, @Neil_P._Quinn_WMF – thank you. The events should be logged as follows...

The following events should should live in Schema:VisualEditorFeaureUse:
- When a contributors taps “<” in “Save your changes” dialog
- When a contributors taps “<” in “Review your changes” view
- When a contributor taps to "Preview"/"Review" their changes
- When a contributor taps "Visual" or "Wikitext" in “Review your changes” view
- When a contributor taps the "This is a minor edit" checkbox in the Save dialog
- When a contributor taps the "Watch this page" checkbox in the Save dialog
- When a contributor taps the “>” in v1 of the toolbar

The following events should should live in Schema:EditAttemptStep
- "When a contributor "starts" an edit"

Neil_P._Quinn_WMF added a project: Product-Analytics.Sep 6 2019, 4:30 PM

Neil_P._Quinn_WMF moved this task from Triage to Tracking on the Product-Analytics board.

@Neil_P._Quinn_WMF / @MNeisler : Given that the "starts an edit" part requires a change to the EditAttemptStep schema, could you verify that all I'd need to do is add, say, "firstTransaction" to the action enum, and add a first_transaction_timing integer? No other extra data required?

Change 535218 had a related patch set uploaded (by DLynch; owner: DLynch):
[mediawiki/extensions/VisualEditor@master] MWSaveDialog: add more instrumentation

https://gerrit.wikimedia.org/r/535218

gerritbot added a project: Patch-For-Review.Sep 9 2019, 3:53 PM

When a contributors taps...

“<” in “Save your changes” dialog	added in patch: mwSave.dialog-abort
“<” in “Review your changes” view	added in patch: mwSave.dialog-review
to "Preview"/"Review" their changes	added in patch: mwSave.dialog-review/preview
"Visual" or "Wikitext" in “Review your changes” view	added in patch: mwSave.review-switch-source / mwSave.review-switch-visual
the "This is a minor edit" checkbox in the Save dialog	added in patch: mwSave.checkbox-wpMinoredit
the "Watch this page" checkbox in the Save dialog	added in patch: mwSave.checkbox-wpWatchthis
the “>” in v1 of the toolbar	already present in EditAttemptStep as saveIntent

@ppelberg do you actually want that last one duplicated?

DLynch moved this task from Ready for Development to Code review on the VisualEditor (Current work) board.Sep 9 2019, 3:53 PM

In T229079#5475532, @DLynch wrote:

When a contributors taps...

“<” in “Save your changes” dialog added in patch: mwSave.dialog-abort

“<” in “Review your changes” view added in patch: mwSave.dialog-review

to "Preview"/"Review" their changes added in patch: mwSave.dialog-review/preview

"Visual" or "Wikitext" in “Review your changes” view added in patch: mwSave.review-switch-source / mwSave.review-switch-visual

the "This is a minor edit" checkbox in the Save dialog added in patch: mwSave.checkbox-wpMinoredit

the "Watch this page" checkbox in the Save dialog added in patch: mwSave.checkbox-wpWatchthis

the “>” in v1 of the toolbar already present in EditAttemptStep as saveIntent

@ppelberg do you actually want that last one duplicated?

@DLynch, I hadn't realized saveIntent mapped to a contributing tapping the “>” button in the toolbar (see below). In which case, "no," I see no reason to create a new/another event for one that already exists. I appreciate you checking.

In T229079#5475160, @DLynch wrote:

@Neil_P._Quinn_WMF / @MNeisler : Given that the "starts an edit" part requires a change to the EditAttemptStep schema, could you verify that all I'd need to do is add, say, "firstTransaction" to the action enum, and add a first_transaction_timing integer? No other extra data required?

@DLynch - Yes, that seems right. No extra data should be required.

@MNeisler - Sorry, I didn't think to ask -- for the timing, would it be more helpful if I stored time-since-initialization or time-since-ready? That'd sort of be whether this event is conceptually grouped with the editor-loading cluster or the saving cluster.

I have updated the schema for now, with time-since-ready: https://meta.wikimedia.org/w/index.php?title=Schema:EditAttemptStep&diff=19364181&oldid=19273145

The answer to that question will just require a documentation change if I picked incorrectly.

Change 535319 had a related patch set uploaded (by DLynch; owner: DLynch):
[mediawiki/extensions/WikimediaEvents@master] Update EditAttemptStep

https://gerrit.wikimedia.org/r/535319

Change 535320 had a related patch set uploaded (by DLynch; owner: DLynch):
[mediawiki/extensions/VisualEditor@master] Add new EditAttemptStep stage: firstTransaction

https://gerrit.wikimedia.org/r/535320

Change 535321 had a related patch set uploaded (by DLynch; owner: DLynch):
[mediawiki/extensions/MobileFrontend@master] Add new EditAttemptStep stage: firstTransaction

https://gerrit.wikimedia.org/r/535321

Okay! That cluster of patches implements the firstTransaction step on desktop and mobile. It logs it with the timing as "time since the document was ready".

Change 535319 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Update EditAttemptStep

https://gerrit.wikimedia.org/r/535319

ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.22; 2019-09-10).Sep 10 2019, 1:00 AM

I assume this should be tracked in non-VE editors as well. In which case "firstTransaction" might be renamed "firstEdit" or "firstChange" as non VE surfaces don't necessarily use transactions. We should also write patches for those editors.

We should also write patches for those editors.

I did have that as a to-do, just lower priority than the initial round of VE testing.

Change 535599 had a related patch set uploaded (by DLynch; owner: DLynch):
[mediawiki/extensions/WikimediaEvents@master] Update EditAttemptStep version

https://gerrit.wikimedia.org/r/535599

Change 535599 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Update EditAttemptStep version

https://gerrit.wikimedia.org/r/535599

Change 535612 had a related patch set uploaded (by DLynch; owner: DLynch):
[mediawiki/extensions/WikiEditor@master] Implement firstChange logging

https://gerrit.wikimedia.org/r/535612

Okay, current patches implement firstChange in: desktop VisualEditor, desktop WikiEditor, MobileFrontend wikitext and visual.

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.23; 2019-09-17); removed MW-1.34-notes (1.34.0-wmf.22; 2019-09-10).Sep 10 2019, 3:00 PM

Change 535321 merged by jenkins-bot:
[mediawiki/extensions/MobileFrontend@master] Add new EditAttemptStep stage: firstChange

https://gerrit.wikimedia.org/r/535321

Change 535320 merged by jenkins-bot:
[mediawiki/extensions/VisualEditor@master] Add new EditAttemptStep stage: firstChange

https://gerrit.wikimedia.org/r/535320

Change 535612 merged by jenkins-bot:
[mediawiki/extensions/WikiEditor@master] Implement firstChange logging

https://gerrit.wikimedia.org/r/535612

ppelberg added a subtask: T232790: Verify editing flow events are firing properly.Sep 12 2019, 10:23 PM

ppelberg merged a task: T221321: Instrument "refreshed" toolbar.Sep 13 2019, 11:57 PM

Change 535218 merged by jenkins-bot:
[mediawiki/extensions/VisualEditor@master] MWSaveDialog: add more instrumentation

https://gerrit.wikimedia.org/r/535218

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.24; 2019-09-24); removed MW-1.34-notes (1.34.0-wmf.23; 2019-09-17).Sep 18 2019, 4:00 PM

T232790 is basically the QA process, after all.

ppelberg mentioned this in T232790: Verify editing flow events are firing properly.Sep 19 2019, 9:23 PM

ppelberg mentioned this in T233823: Mobile VE edit flow: instrumentation post-deployment data checks.Sep 25 2019, 3:25 PM

dchan moved this task from Current work to Analysis on the VisualEditor board.Oct 6 2019, 6:09 AM

dchan edited projects, added VisualEditor; removed VisualEditor (Current work).

It looks like this is ready for Peter's review

ppelberg closed this task as Resolved.Oct 23 2019, 1:10 AM

Instrument each step of mobile VE edit flowClosed, ResolvedPublicActions