Page MenuHomePhabricator

Instrument mobile wikitext editor fallback worfklow
Closed, ResolvedPublic

Description

Background

The deployment of T239676 will introduce an option for all contributors to opt into the mobile wikitext editor, even in instances when the mobile visualeditor would have otherwise loaded by default. [1]

To ensure these contributors are appropriately accounted for in our analysis, we will need to add new tracking events that enable us to see which users opted into the mobile wikitext editor by way of the new "fallback workflow" T239676 will introduce.

Instrumentation requirements

Instrumentation to be added to VisualEditorFeatureUse. Proposed event names are below:
Event: A contributor is shown the fallback prompt
VEFU feature: mobileVisualFallback
VEFU action: context-show

Event: A contributor elects to use the fallback editor
VEFU feature:mobileVisualFallback
VEFU action: fallback-confirm

What we would like to know

We would like to know...

  • When/whether a contributor is shown the fallback prompt
  • Whether they elect to use the fallback editor

Done

  • The instrumentation required to satisfy the "What we would like to know" requirements have been defined
  • The instrumentation required to satisfy the "What we would like to know" requirements have been implemented

  1. E.g. a contributor in the A/B test could be assigned to the default-visual bucket, but elect to use the mobile wikitext editor if they felt the mobile visual editor was taking too long to load.

Event Timeline

I noted in the commit message of https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/+/554222:

We […] log an init event with
init_mechanism=tooslow.

Is this enough? If not, what would you like?

I noted in the commit message of https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/+/554222:

We […] log an init event with
init_mechanism=tooslow.

Is this enough? If not, what would you like?

This instrumentation should be helpful – thank you, Bartosz. I've added what we would like to know about how people engage with the fallback experience to the task description.

If you have thoughts about either of the above, please add them here.

Next steps

  • @Mayakp.wiki to review "What we would like to know" section in the task description and determine what – if any – additional instrumentation we will need to implement.
    • Note: I do not anticipate Maya having time to do this for two weeks, considering all we have going on with the instrumentation around the Replying workflow (T243364)
MNeisler triaged this task as Medium priority.Sep 18 2023, 2:18 PM
MNeisler edited projects, added Product-Analytics (Kanban); removed Product-Analytics.

Here are two proposed events to be logged in VisualEditorFeatureUse to track engagement with this feature

Event: A contributor is shown the fallback prompt
VEFU feature: mobileVisualFallback
VEFU action: context-show

Event: A contributor elects to use the fallback editor
VEFU feature:mobileVisualFallback
VEFU action: fallback-confirm

Note: Once instrumented in VEFU, this data can be joined with EditAttemptStep to allow us to track other session and user-level editing data such as if the contributor published or abandoned their edit after being shown or selecting the fallback editor and associated load times, which might be valuable in future analyses.

MNeisler updated the task description. (Show Details)
MNeisler updated the task description. (Show Details)
MNeisler edited projects, added Product-Analytics; removed Product-Analytics (Kanban).

Change 966289 had a related patch set uploaded (by DLynch; author: DLynch):

[schemas/event/secondary@master] Add a new init_mechanism to editattemptstep

https://gerrit.wikimedia.org/r/966289

Change 966292 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/WikimediaEvents@master] Bump EditAttemptStep to 2.0.1

https://gerrit.wikimedia.org/r/966292

Change 966295 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/MobileFrontend@master] Add instrumentation for slow editor fallback

https://gerrit.wikimedia.org/r/966295

Change 966289 merged by jenkins-bot:

[schemas/event/secondary@master] Add a new init_mechanism to editattemptstep

https://gerrit.wikimedia.org/r/966289

Change 966295 merged by jenkins-bot:

[mediawiki/extensions/MobileFrontend@master] Add instrumentation for slow editor fallback

https://gerrit.wikimedia.org/r/966295

Change 966292 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Bump EditAttemptStep to 2.0.1

https://gerrit.wikimedia.org/r/966292

ppelberg edited projects, added Editing QA; removed Product-Analytics.
ppelberg moved this task from Code Review to QA on the Editing-team (Kanban Board) board.
ppelberg added subscribers: EAkinloose, Ryasmeen, DLynch.

Adding this to Editing QA for @EAkinloose / @Ryasmeen to confirm client-side logging is working as expected.

Note: server-side QA can happen in parallel in T243648.

ppelberg claimed this task.