Page MenuHomePhabricator
Create Task
Maniphest T252930

Post-deployment QA: Reply tool v2.0 instrumentation (DB)
Closed, ResolvedPublic
Actions

Description

T252924 added new fields to the the VisualEditorFeatureUse schema [1][2].

This task is about ensuring these newly added events and fields are being logged correctly in database.

Events and fields to check

@Mayakp.wiki to link to the instrumentation sheet once she and @ppelberg finalize it.

Interfaces/integrations to check

  • Mobile Frontend
  • Reply tool
  • VisualEditor
  • WikiEditor

Timing

Work on this task can begin once T251654 has been resolved.

Done

  • Document whether the events defined in the instrumentation spec [3] are landing the database in the ways we expect
  1. https://www.mediawiki.org/wiki/VisualEditor/FeatureUse_data_dictionary
  2. https://meta.wikimedia.org/wiki/Schema:VisualEditorFeatureUse
  3. https://docs.google.com/spreadsheets/d/1iMWKmT7nHimjaa7OItGINoTVGtnaIy86I3ICj0RN5KY/edit#gid=2101596546

Related Objects
Search...

Event Timeline

ppelberg created this task.May 15 2020, 11:18 PM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptMay 15 2020, 11:18 PM
ppelberg mentioned this in T244498: Replies v2.0: determine what additional instrumentation is needed.May 15 2020, 11:21 PM
ppelberg added a parent task: T252931: Instrument Reply tool v2.0.May 15 2020, 11:39 PM
ppelberg mentioned this in T252927: Post-deployment QA: Reply tool v2.0 instrumentation.May 19 2020, 1:35 AM
ppelberg added a comment.EditedMay 19 2020, 1:41 AM

Task description update
Making the following updates to the task description following a chat with @Mayakp.wiki:

  • ADDED: === Interfaces/integrations to check: Reply tool, WikiEditor, VisualEditor, Mobile Frontend
ppelberg updated the task description. (Show Details)May 19 2020, 1:42 AM
ppelberg updated the task description. (Show Details)
ppelberg moved this task from Needs Discussion / Investigation to Upcoming on the Editing-team (Q3 2019-2020 Kanban Board) board.Jun 3 2020, 2:39 PM
ppelberg moved this task from Needs Prioritization to Required after release on the OWC2020 (OWC2020 Replying 2.0) board.Jun 3 2020, 2:52 PM
ppelberg mentioned this in T254291: Add additional events to VisualEditorFeatureUse.Jun 15 2020, 10:08 PM
ppelberg edited projects, added: Editing-team (Tracking); removed: Editing-team (Q3 2019-2020 Kanban Board).Jun 17 2020, 6:56 PM
ppelberg moved this task from Backlog to Analytics on the Editing-team (Tracking) board.
Mayakp.wiki added a project: Product-Analytics.Jun 22 2020, 4:15 PM
LGoto assigned this task to Mayakp.wiki.Jun 30 2020, 7:48 PM
LGoto triaged this task as Medium priority.
LGoto edited projects, added: Product-Analytics (Kanban); removed: Product-Analytics.
ppelberg moved this task from Required after release to Required for Release on the OWC2020 (OWC2020 Replying 2.0) board.Jul 3 2020, 4:49 PM
ppelberg edited projects, added: OWC2020 (Reply Tool Opt Out); removed: OWC2020 (OWC2020 Replying 2.0).Jul 27 2020, 11:25 PM
ppelberg moved this task from Needs Prioritization to Required for Release on the OWC2020 (Reply Tool Opt Out) board.
Mayakp.wiki mentioned this in T258761: Test VisualEditorFeatureUse schema changes in Reply Tool.Jul 29 2020, 4:55 PM
ppelberg updated the task description. (Show Details)Jul 29 2020, 10:24 PM

Task description update
Per today's conversation with @Mayakp.wiki, I've added the below to the task description's "Events and fields to check" section:

ppelberg added a comment.Aug 4 2020, 11:24 PM

Status update
This task is unblocked now that T252925 is resolved.

ppelberg mentioned this in T257501: Add event for logging when people are prevented from switching from source to visual.Aug 4 2020, 11:57 PM
ppelberg updated the task description. (Show Details)Aug 5 2020, 7:13 PM
Mayakp.wiki moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Aug 12 2020, 5:39 PM
Mayakp.wiki added a comment.Sep 1 2020, 10:28 PM

Completed Data-QA. Verdict: Data in both the schemas look good!
Here are some notes/observations.

Part 1 : Discussion Tools QA

  • All the Discussion Tools data in EditAttemptStep and VisualEditorFeatureUse looks good
  • saveFailure and Abort events are quite low. Overall the tool seems to have a good success ratio
  • Discussion Tools are deployed to 12 wikis currently : Deployment Status
  1. Arabic
  2. Dutch
  3. French
  4. Hungarian
  5. Chinese
  6. Czech
  7. Serbian
  8. Georgian
  9. Sorani Kurdish
  10. Swedish
  11. Catalan
  12. Korean
  • However, we do see DT events for other wikis, but the number of events in those wikis are few in number and hence not a big issue.
  • As expected, there are no events in any other platform other than "Desktop" since Replying feature is currently deployed only to desktop
  • Since Replying is an Opt-in Beta Feature currently, there are no events with logged out users
  • Looking at the edit count bucket, it seems like a majority of the users trying the Reply tool are experienced editors
  • On a given day in all the wikis, users are not switching between source mode and visual mode a whole lot
  • Overall it seems like users prefer to tag someone in a conversation by Typing @ in the Reply tool's visual mode (inside the text box) instead of clicking the icon "👤 " from the visual mode's toolbar
  • Very few users who have opted to "Watch" a Discussion are using the Checkbox for watching a page
  • Overall we do not see a lot of Styling changes made by users. This could be because this is a Talk page edit and not an Article edit and doesnt need a whole lot of styling.

Part 2 : Add fields to VisualEditorFeatureUse

  • Overall, data for new fields added in VisualEditorFeatureUse looks good
  • Looked at data from 07-01-2020 and found a huge number of Null 'editor_Interface' records for Mobile platform. This bug was identified during Instrumentation QA in T252925 Issue #1
  • Verified that these null records have been reducing since the fix applied on Aug 4, 2020 and assigned a valid editor_interface.
  • Observed a few sessions in VisualEditorFeatureUse schema (that have null editor_interface) that do not exist in EditAttempStep. We are not sure of the exact cause of this, but this may not be an issue as the number of events with null editor_interface is extremely low from Aug 4, 2020 onwards (almost 0 now).
Mayakp.wiki moved this task from Doing to Needs Review on the Product-Analytics (Kanban) board.Sep 1 2020, 10:34 PM
Mayakp.wiki added a subscriber: MNeisler.

Status and Scenarios tested :
Replying QC Template Post Deployment Data QA
Link to noteboks-
Data QA_Part 1_DiscussionTools
Data QA_Part 2_VEfu-new-fields

@MNeisler : can you pls review the data-QA notebooks and let me know if I've missed any relevant data checks? Thanks for your help with this ! :)

MNeisler closed this task as Resolved.Sep 15 2020, 2:53 PM

@Mayakp.wiki - I completed my review of your data-QA notebooks. Everything looks good and you covered all the relevant data checks so no additional changes or checks needed. Thank you for your work on this!

Documenting a few additional observations and updates made during my review below:

Looking at the edit count bucket, it seems like a majority of the users trying the Reply tool are experienced editors

  • The predominant use by experienced editors is likely due to the reply tool only being available as an opt-in feature during the time period reviewed. I'd expect a higher usage among junior contributors as it is released as an opt-out feature.

Overall it seems like users prefer to tag someone in a conversation by Typing @ in the Reply tool's visual mode (inside the text box) instead of clicking the icon "👤 " from the visual mode's toolbar

It looks like only slightly though - As of September 14th, there were 274 events created by typing @ in the Reply Tool and 201 events where the user clicked the icon to tag someone in a conversation. The number of daily events recorded for each seems to be stabilizing as well.

Verified that these null records have been reducing since the fix applied on Aug 4, 2020 and assigned a valid editor_interface.

  • Did a quick check of recent data. There's still an occasional event being recorded as NULL 'editor_interface' on mobile but the number of NULL values has remained significantly low (only 1 or 3 events every few days) following the August 4th fix and the majority of mobile events have been assigned visualeditor/wikitext editor values.

Observed a few sessions in VisualEditorFeatureUse schema (that have null editor_interface) that do not exist in EditAttempStep. We are not sure of the exact cause of this, but this may not be an issue as the number of events with null editor_interface is extremely low from Aug 4, 2020 onwards (almost 0 now).

I reran the query to look at data from August 4th (after the null editor_interface fix was deployed) to the end of August and there are only 30 sessions in VEFU that are not present in EditAttemptStep, which is a significant drop from the 2023 sessions originally noted when August 3rd events were included in the query. Based on this, it looks like this issue was also addressed by the Aug 4th fix.

@ppelberg - Going ahead and marking this as resolved as I confirmed the data looks as expected but let us know if you have any questions.

Mayakp.wiki added a comment.Sep 15 2020, 10:56 PM

thank you @MNeisler ! for your review :)

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits