Page MenuHomePhabricator
Create Task
Maniphest T344465

Instrument edit notices (desktop + mobile)
Closed, ResolvedPublic
Actions

Assigned To
ppelberg
Authored By
ppelberg
Aug 17 2023, 9:37 PM
Tags
Referenced Files
F38214168: image.png
Oct 12 2023, 1:05 AM
F38214155: image.png
Oct 12 2023, 1:05 AM
F38214151: image.png
Oct 12 2023, 1:05 AM
F38213808: Screenshot 2023-10-11 at 4.21.29 PM.png
Oct 11 2023, 11:32 PM
Subscribers
Aklapper
AlexisJazz
DLynch
MNeisler
ppelberg
Ryasmeen
Xaosflux

Description

This task involves the work of implementing the instrumentation necessary to report on the metrics necessary to the evaluate the impact of introducing mobile edit notices. Note: the actual analysis will happen in T313279.

Requirements

  1. Instrumentation is added that enables us to report on metrics like:
    • How does the rate at which people complete the edits they start on mobile vary between the time before mobile edit notices were introduced and the time after mobile edit notices were introduced?
    • How does the rate at which edits are reverted on mobile vary between the time before mobile edit notices were introduced and the time after mobile edit notices were introduced?
    • How does the rate at which people abandon the edit sessions they initiate on mobile vary between the time before mobile edit notices were introduced and the time after mobile edit notices were introduced?
  2. We should be able to group the metrics listed in "1." by the experience level of the people making edits as well as the editing interface/mode (source / visual) they were using

QA

  • Editing QA to verify client-side logging being emitted as expected
  • @MNeisler to verify event appearing server-side as expected

Details

SubjectRepoBranchLines +/-
Track when edit notices are shownmediawiki/extensions/MobileFrontendmaster+8 -2
Track when edit notices are shownmediawiki/extensions/VisualEditormaster+2 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

ppelberg created this task.Aug 17 2023, 9:37 PM
ppelberg updated the task description. (Show Details)
ppelberg added a project: Goal.
ppelberg moved this task from Incoming to Ready to Be Worked On on the Editing-team (Kanban Board) board.Aug 23 2023, 12:06 AM
gerritbot added a comment.Aug 24 2023, 7:04 PM

Change 952262 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/MobileFrontend@master] Track when edit notices are shown

https://gerrit.wikimedia.org/r/952262

gerritbot added a project: Patch-For-Review.Aug 24 2023, 7:04 PM

Change 952263 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@master] Track when edit notices are shown

https://gerrit.wikimedia.org/r/952263

DLynch claimed this task.Aug 24 2023, 7:06 PM

Those patches will log when the edit notices are shown on desktop (VE / NWE) and mobile. Logged in VisualEditorFeatureUse as feature: notices, action: show.

ppelberg moved this task from Ready to Be Worked On to Code Review on the Editing-team (Kanban Board) board.Aug 24 2023, 10:52 PM
ppelberg renamed this task from Instrument mobile edit notices to Instrument edit notices (desktop + mobile).Aug 28 2023, 5:51 AM
MNeisler moved this task from Triage to Tracking on the Product-Analytics board.Sep 5 2023, 3:07 PM
ppelberg added a parent task: T312587: Show edit notices within mobile editing interfaces.Sep 6 2023, 4:00 PM
Esanders moved this task from Code Review to QA on the Editing-team (Kanban Board) board.Sep 8 2023, 2:44 PM
Esanders added a project: Editing QA.
gerritbot added a comment.Sep 8 2023, 3:05 PM

Change 952263 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Track when edit notices are shown

https://gerrit.wikimedia.org/r/952263

gerritbot added a comment.Sep 8 2023, 3:11 PM

Change 952262 merged by jenkins-bot:

[mediawiki/extensions/MobileFrontend@master] Track when edit notices are shown

https://gerrit.wikimedia.org/r/952262

Maintenance_bot removed a project: Patch-For-Review.Sep 8 2023, 3:30 PM
ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.26; 2023-09-12).Sep 8 2023, 4:00 PM
MNeisler added a comment.Sep 12 2023, 2:28 PM

The instrumentation deployed for this (feature: notices, action: show in VEFU) looks good to me and will be sufficient to calculate the metrics identified in the requirements of the task and analysis.

Just a quick note/reminder that to calculate the revert rate we'll need to join VEFU/EditAttemptStep data to mediawiki_history using the published edits' revision_id, which is only retained for 90 days in EditAttemptStep. As a result, analyses on that metric will only be able to look at the last 90 days.

ppelberg updated the task description. (Show Details)Sep 13 2023, 8:39 PM
ppelberg added a comment.Sep 19 2023, 11:54 PM

Documenting what @MNeisler talked about offline on 13 Sep...

In T344465#9160414, @MNeisler wrote:

The instrumentation deployed for this (feature: notices, action: show in VEFU) looks good to me and will be sufficient to calculate the metrics identified in the requirements of the task and analysis.

Excellent.

Just a quick note/reminder that to calculate the revert rate we'll need to join VEFU/EditAttemptStep data to mediawiki_history using the published edits' revision_id, which is only retained for 90 days in EditAttemptStep. As a result, analyses on that metric will only be able to look at the last 90 days.

Great spot and understood.

ppelberg moved this task from Inbox to High Priority on the Editing QA board.Sep 20 2023, 5:21 PM
Ryasmeen subscribed.Sep 25 2023, 10:52 PM

Client-side logging is being emitted as expected on Desktop for logged in users with VisualEditorFeatureUse as feature: notices, action: show. I will test the mobile part once this task is resolved T312587.

ppelberg moved this task from QA to Blocked / Needs More Work on the Editing-team (Kanban Board) board.EditedSep 25 2023, 10:53 PM
In T344465#9197708, @Ryasmeen wrote:

Client-side logging is being emitted as expected on Desktop for logged in users with VisualEditorFeatureUse as feature: notices, action: show.

Excellent.

I will test the mobile part once this task is resolved T312587.

Sounds great; moving to "Blocked / Needs more work" to remind us of this.

Ryasmeen added a comment.Oct 11 2023, 11:32 PM

@DLynch: I am not seeing any Client-side logging on Mobile in production for logged in users with VisualEditorFeatureUse as feature: notices, action: show.

Screenshot 2023-10-11 at 4.21.29 PM.png (1×2 px, 766 KB)

DLynch added a comment.Oct 12 2023, 1:05 AM

@Ryasmeen You're testing on enwiki, so could you verify that you've got the EditNoticesOnMobile gadget disabled?

image.png (106×1 px, 28 KB)

If it's enabled, it entirely replaces the new built-in feature. Note below that the styling for me is different from yours on both the new feature and the gadget...

The new MobileFrontend noticesThe EditNoticesOnMobile gadget
image.png (816×648 px, 137 KB)
image.png (1×1 px, 244 KB)
Ryasmeen moved this task from Blocked / Needs More Work to Ready for Sign Off on the Editing-team (Kanban Board) board.Oct 12 2023, 6:25 AM
Ryasmeen edited projects, added Verified; removed Editing QA.
In T344465#9244904, @DLynch wrote:

@Ryasmeen You're testing on enwiki, so could you verify that you've got the EditNoticesOnMobile gadget disabled?

image.png (106×1 px, 28 KB)

If it's enabled, it entirely replaces the new built-in feature. Note below that the styling for me is different from yours on both the new feature and the gadget...

The new MobileFrontend noticesThe EditNoticesOnMobile gadget
image.png (816×648 px, 137 KB)
image.png (1×1 px, 244 KB)

Thanks @DLynch! I did have EditNoticesOnMobile gadget enabled. I am now seeing the event being triggered correctly after disabling it.

Ryasmeen mentioned this in T316178: [Config Change] Make upstream mobile edit notice implementation available at all wikis.Oct 12 2023, 10:59 PM
ppelberg reassigned this task from DLynch to MNeisler.Oct 19 2023, 5:00 PM
ppelberg edited projects, added Editing-team (Tracking); removed Editing-team (Kanban Board).

I'm assigning this task over to Megan to verify the newly-instrumented events are being logged as expected server-side.

ppelberg moved this task from Backlog to Analytics on the Editing-team (Tracking) board.Oct 19 2023, 5:01 PM
MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.Oct 20 2023, 4:25 PM
ppelberg added a parent task: T201595: Mobile frontend web does not present edit notices.Oct 23 2023, 5:23 PM
MNeisler reassigned this task from MNeisler to ppelberg.Oct 24 2023, 4:19 PM
MNeisler moved this task from Next 2 weeks to [Deprecated] Done (previously: Needs sign-off) on the Product-Analytics (Kanban) board.

@ppelberg I've checked and confirmed that events are being logged as expected in VisualEditorFeatureUse. See summary of checks and some initial data counts below:

The data below reflects all editing sessions where an edit notice was shown (feature: notices, action: show) from 11 October through today:

  • Events logged on both desktop and mobile web platforms following 11 October deployment:
platformn_sessions
desktop3285046
phone370893
  • Events logged on all namespaces (95% of sessions on the main page namespace (ns:0))
  • Event logged for both logged-in and logged-out users. Edit notices have mostly been shown to unregistered users (92% of editing sessions).
user_statusn_sessions
registered303625
unregistered3352364
  • Events logged for both visual and source modes
editor_interfacen_sessions
visualeditor3493606
wikitext814220
wikitext-201762908
  • Confirmed that associated EditAttemptStep events are logged as expected so we can calculate edit completion and abandonment rate.

Note: I've also updated the VisualEditorFeatureUse dictionary with these new events now that it has been deployed.

ppelberg closed this task as Resolved.Oct 24 2023, 7:50 PM

Looks great; thank you, @MNeisler. Per what we talked about offline today, we'll look at edit completion and abandonment rates in T313279.

AlexisJazz mentioned this in T313279: [Impact Analysis] Evaluate Impact of introducing mobile edit notices.Oct 24 2023, 9:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits