Page MenuHomePhabricator

Mobile VE edit flow: instrumentation post-deployment data checks
Closed, ResolvedPublic

Description

Overview

This task involves the work with making sure we are logging data in such a way that we will be able to track the additional events we implemented to better understand how contributors move through each step of the edit flow.

Information about the additional events we implemented can be found in this task: T229079.

Information about the initial round of QA that was done on this instrumentation can be found in this task: T232790

Additional Instrumentation

See this task's description: T232790

"Done"

  • Product Analytics has verified the events listed in the "Additional instrumentation" section are being logged correctly

All QA checks and comments are provided in this QA document.

Event Timeline

Reassigning to @Mayakp.wiki, our Data Quality Analyst. I'll pair with her to perform the data checks.

Completed one round of testing on the following fields and everything looks okay:

  • dialog-abort
  • review-switch-source
  • review-switch-visual

Need confirmation from editing team on the behavior of the below actions:

  • dialog-approve
  • dialog-review
  • dialog-preview

Editing team fixed the events that were getting fired twice. Will test these next week:

  • checkbox-wpMinoredit
  • checkbox-wpWatchthis

In addition, we would be interested to know since when did these actions start getting fired for mwSave feature :

  • review-initial-source
  • review-initial-visual
  • dialog-report
  • dialog-resolve
  • dialog-save

All the scenarios and testing done are provided in this document
QC Template : Instrumentation post-deployment data checks

Megan will check with Peter tomorrow on the following observations. If we check when the above events started firing it will help to get better clarity.

Need confirmation from editing team on the behavior of the below actions:
dialog-approve
dialog-review
dialog-preview

In addition, we would be interested to know since when did these actions start getting fired for mwSave feature :
review-initial-source
review-initial-visual
dialog-report
dialog-resolve
dialog-save

Will test the following this week:
Editing team fixed the events that were getting fired twice. Will test these this week:
checkbox-wpMinoredit
checkbox-wpWatchthis

@DLynch - In QA, @Mayakp.wiki noticed the following mwSave feature actions were being recorded:

review-initial-source
review-initial-visual
dialog-report
dialog-resolve
dialog-save

These were not initially identified as new actions that were implemented as part of T229079 and are not currently defined in the data dictionary.
Can you confirm if these actions were added as part of the recent instrumentation changes and their purpose? Thanks!

Mayakp.wiki moved this task from Triage to Doing on the Product-Analytics board.

In addition to Megan's comment, @DLynch and @ppelberg can you please also confirm : When does the 'dialog-review' and 'dialog-preview' get fired ?

The following are in Passed status as mentioned in my QA document:
dialog-abort
dialog-approve
review-switch-source
review-switch-visual
checkbox-wpMinoredit
checkbox-wpWatchthis

Awaiting confirmation on the below:

Megan will check with Peter tomorrow on the following observations. If we check when the above events started firing it will help to get better clarity.

Need confirmation from editing team on the behavior of the below actions:
dialog-approve
dialog-review
dialog-preview

In addition, we would be interested to know since when did these actions start getting fired for mwSave feature :
review-initial-source
review-initial-visual
dialog-report
dialog-resolve
dialog-save

Will test the following this week:
Editing team fixed the events that were getting fired twice. Will test these this week:
checkbox-wpMinoredit
checkbox-wpWatchthis

During testing I also found a bunch of other actions being stored under the mwSave feature. We can ignore them if required as the number of times these actions appear is very very small..but it would help to know if its a bug:
review-initial-schnark
checkbox-has
checkbox-intersects
checkbox-unique
checkbox-wpReviewEdit

Mayakp.wiki changed the task status from Open to Stalled.Oct 14 2019, 11:07 PM
Mayakp.wiki changed the task status from Stalled to Open.
Mayakp.wiki moved this task from Doing to Blocked on the Product-Analytics board.

@DLynch - In QA, @Mayakp.wiki noticed the following mwSave feature actions were being recorded:

These are all from the last patch to that ticket, yes.

review-initial-source
review-initial-visual

These are both the mode of the review dialog which is initially selected when it opens, and should always immediately follow a dialog-review. Since it remembers the previous mode, otherwise you wouldn't be able to tell -- this seemed like a logical compliment to "When a contributor taps "Visual" or "Wikitext" in “Review your changes” view", without implying that a tap had actually taken place.

dialog-report
dialog-resolve
dialog-save

These are all side-effects of the "When a contributor taps to "Preview"/"Review" their changes" item from that ticket. The dialog- actions all come from the code, and just log whatever the dialog was told rather than being explicitly written into the logging. Thus dialog-report is when someone clicks the "report incorrect display for this change" link, dialog-resolve would be in the dialog's conflict resolution mode, and dialog-save would happen when someone clicks the "publish changes" button (and thus be redundant with EditAttemptStep's saveAttempt).

Also, for reference though it wasn't asked about, dialog-approve is when you switch back to the main save panel with the summary after viewing one of the others.

During testing I also found a bunch of other actions being stored under the mwSave feature. We can ignore them if required as the number of times these actions appear is very very small..but it would help to know if its a bug:
review-initial-schnark
checkbox-has
checkbox-intersects
checkbox-unique
checkbox-wpReviewEdit

Those are all probably interactions with other code. The logging we do just does logging based on the ids that are present in the code. If other extensions / gadgets / userscripts insert things, they'll get included in the logging. wpReviewEdit is from the FlaggedRevs extension, for instance.

For that first one, @Schnark is a contributor and may have been doing something with gadgets or userscripts to trigger that.

Sorry, just to be explicit:

When does the 'dialog-review' and 'dialog-preview' get fired ?

When you click the "review your changes" or "show preview" buttons that are at the bottom of the save dialog.

review-initial-schnark

For that first one, @Schnark is a contributor and may have been doing something with gadgets or userscripts to trigger that.

My user script https://de.wikipedia.org/wiki/Benutzer:Schnark/js/diff adds its own diff mode, and integrates this into different places, including VE's save dialog. So review-initial-schnark is just like the other review-initial-*, but with the diff mode from my script.

Thanks @DLynch for that explanation. It was really helpful ! and I now have much better clarity.

Mayakp.wiki closed this task as Resolved.EditedOct 22 2019, 10:16 PM

All checks completed. Data for the instrumented Actions is getting logged correctly in VisualEditorFeatureUse (for mwSave feature) and EditAttemptStep schemas. Marking this ticket as resolved.
Will post notebook link later this week.