Ensure logging is in place to compare MobileFrontend and DiscussionTools new topic and new comment completion rates
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• ppelberg
	Feb 18 2022, 7:27 PM

Description

Part of the work we will be doing to evaluate the impact of the collection of improvements the Editing Team will be making to mobile talk pages (T278588) will involve conducting two impact analyses (T298058 + T298065).

In order to conduct these analyses, we will need to ensure sufficient instrumentation is in place to compare the DT-enhanced version of mobile web talk pages to the existing, Read as wiki page and MobileFrontend talk overlay, versions of mobile web talk pages.

Requirements

The MobileWebUIActions schema is logging the following events within the MobileFrontend's replying workflow:
- When a person taps "into" the reply input field
- When a person enters a character into the reply input field
- When a person publishes a comment they drafted
The MobileWebUIActions schema is logging the following events within the MobileFrontend's new topic workflow:
- When a person initiates the workflow by way of tapping the Add topic button
- When a person enters text into either of the new topic workflow's fields (Subject or Make a suggestion or voice a concern)
- When a person publishes a topic they drafted

NOTE: the answers to ===Open Questions #1 and/or #2 below may mean no additional work is required to achieve some, if not all, of the requirements listed above.

Testing

action	uiactions event	newly added?	Client-side logging verified?
When a person taps "into" the reply input field	talkpage.focus-comment	Yes	Yes
When a person enters a character into the reply input field	talkpage.input-comment	Yes	Yes
When a person publishes a comment they drafted	talkpage.publish-comment	Yes	Yes
When a person initiates the workflow by way of tapping the Add topic button	talkpage.add-topic		Yes
When a person enters text into either of the new topic workflow's fields	talkpage.input-topic	Yes	Yes
When a person publishes a topic they drafted	talkpage.publish-topic	Yes	Yes

Open Questions

1. What – if any – events within the MobileFrontend's new topic workflow are currently being logged?
- See T302108#8194966.
2. What – if any – events within the MobileFrontend's replying workflow

are currently being logged?

See T302108#8194966.

Done

Answers to all === Open Questions are documented
All ===Requirements are met
@Ryasmeen / @EAkinloose verify clients are emitting the newly-added events listed in the ===Testing section above
@MNeisler verifies the newly-added events are being logged in the DB as expected

i. @DLynch surfaced this idea during the Editing Team's 16 February 2022 team meeting (see Editing Scratch Archive for notes).

Details

Subject	Repo	Branch	Lines +/-
MobileWebUIActions sample rate to 1 on testwiki	operations/mediawiki-config	master	+1 -0
Track various talk overlay events via UIActions	mediawiki/extensions/MobileFrontend	master	+16 -4
Let uiactions logging be triggered via mw.track	mediawiki/extensions/WikimediaEvents	master	+10 -0

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		matmarex	T319145 Remove MobileFrontend/Minerva code for talk page tools
Resolved		Jdlrobson	T312312 Deprecate MobileFrontend's "About this page" functionality on talk pages
Resolved		None	T278588 Mobile talk page improvements
Resolved	BUG REPORT	matmarex	T326080 User talk page seems to be empty in mobile
Resolved		matmarex	T327047 English Wikipedia WikiProject banners no longer visible on mobile
Resolved		None	T298060 [RELEASE TICKET] Offer Mobile DiscussionTools at All Wikis
Resolved		• ppelberg	T298062 [A/B Test] Run an A/B test to evaluate impact of mobile DiscussionTools
Resolved		• ppelberg	T302108 Ensure logging is in place to compare MobileFrontend and DiscussionTools new topic and new comment completion rates
Resolved		MNeisler	T301026 Create instrumentation spec for mobile DiscussionTools
Duplicate		MNeisler	T303653 [SPIKE] Investigate how sessionID is used in various mobile talk page schemas
Resolved		DLynch	T303654 [SPIKE] Investigate practicality of making sampling rate consistent across mobile talk page schemas
Resolved		• ppelberg	T307640 Add page view token to UIAction schemas

Event Timeline

• ppelberg created this task.Feb 18 2022, 7:27 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 18 2022, 7:27 PM

• ppelberg moved this task from Incoming to Upcoming on the Editing-team (Kanban Board) board.Feb 18 2022, 7:27 PM

• ppelberg added a parent task: T282638: [RELEASE TICKET] Offer Mobile Reply and New Discussion Tools at Partner Wikis.

I'll note that a maybe-important aspect that'd be hard to compare would be "edits that aren't even attempted" in the current mobile overlay -- because with DT we're adding a new ability to reply to nested comments, and so there's presumably people who would have previously bounced off the page without leaving any trace on EditAttemptStep who will now instead make an edit.

• ppelberg moved this task from Upcoming to Ready to Be Worked On on the Editing-team (Kanban Board) board.Feb 21 2022, 10:37 PM

• ppelberg moved this task from Backlog to Triaged on the DiscussionTools board.Feb 24 2022, 4:11 PM

• ppelberg added a subtask: T301026: Create instrumentation spec for mobile DiscussionTools.Mar 3 2022, 6:49 PM

MNeisler mentioned this in T301026: Create instrumentation spec for mobile DiscussionTools.Mar 9 2022, 7:27 PM

Per what @DLynch, @MNeisler, and I talked about offline last week (18 May 2022):

We DECIDED that rather than adding EditAttemptStep schema events to the existing MobileFrontend talk page overlay, we will instead verify/ensure that the MobileFrontend includes sufficient logging for us to compare when people start and publish an edit using its existing new topic and commenting workflows
We DECIDED that for the purpose of comparing the edit completion rates between the Reply Tool and the existing MobileFrontend commenting experience, we will consider the denominator in the edit completion rate to be the firstChange event within the Reply Tool/EditAttemptStep schema.
In line with "1." and "2." we're going to reframe this task as being about investigating what - if any – instrumentation needs to be added to the existing MobileFrontend new topic and commenting experiences in order to compare the rates at which people complete the comments and new topics they start with the MobileFrontend talk tools and the mobile Discussion Tools.

• ppelberg renamed this task from [SPIKE] Investigate the value and work involved with adding EditAttemptStep to the mobile talk overlay to Ensure logging is in place to compare MobileFrontend and DiscussionTools new topic and new comment completion rates.May 26 2022, 3:07 AM

• ppelberg updated the task description. (Show Details)

• ppelberg added a project: Goal.May 27 2022, 10:09 PM

• ppelberg assigned this task to DLynch.Jul 5 2022, 5:15 PM

• ppelberg closed subtask T301026: Create instrumentation spec for mobile DiscussionTools as Resolved.Aug 10 2022, 8:20 PM

• ppelberg added a parent task: T298062: [A/B Test] Run an A/B test to evaluate impact of mobile DiscussionTools.Aug 17 2022, 8:13 PM

• ppelberg added a project: TPP-Phase1.

• ppelberg moved this task from Tickets for Consideration to Mobile on the TPP-Phase1 board.

MNeisler mentioned this in T298058: [Impact Analysis] Evaluate Impact of Mobile Reply and New Discussion Tools.Aug 24 2022, 6:23 PM

Change 827530 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/WikimediaEvents@master] Let uiactions logging be triggered via mw.track

https://gerrit.wikimedia.org/r/827530

Change 827531 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/MobileFrontend@master] Track various talk overlay events via UIActions

https://gerrit.wikimedia.org/r/827531

action	uiactions event	newly added?
When a person taps "into" the reply input field	talkpage.focus-comment	Y
When a person enters a character into the reply input field	talkpage.input-comment	Y
When a person publishes a comment they drafted	talkpage.publish-comment	Y
When a person initiates the workflow by way of tapping the Add topic button	talkpage.add-topic
When a person enters text into either of the new topic workflow's fields	talkpage.input-topic	Y
When a person publishes a topic they drafted	talkpage.publish-topic	Y

Change 827530 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Let uiactions logging be triggered via mw.track

https://gerrit.wikimedia.org/r/827530

ReleaseTaggerBot added a project: MW-1.39-notes (1.39.0-wmf.27; 2022-08-29).Aug 30 2022, 12:00 AM

Change 827531 merged by jenkins-bot:

[mediawiki/extensions/MobileFrontend@master] Track various talk overlay events via UIActions

https://gerrit.wikimedia.org/r/827531

Maintenance_bot removed a project: Patch-For-Review.Aug 30 2022, 10:30 PM

• ppelberg updated the task description. (Show Details)Sep 2 2022, 10:34 PM

• ppelberg updated the task description. (Show Details)

• ppelberg added subscribers: EAkinloose, Ryasmeen.

It occurs to me to clarify: I interpreted both of the "enters text" requirements as requiring that an event is fired only the first time this happens, rather than for every single keystroke.

In T302108#8209419, @DLynch wrote:

It occurs to me to clarify: I interpreted both of the "enters text" requirements as requiring that an event is fired only the first time this happens, rather than for every single keystroke.

This sounds okay to me considering that in the analysis that depends on this instrumentation (T298062) we'll be looking at edit completion rate which – I do not think – would require us to look at anything more detailed than the first time someone entered text into the Reply or New Topic Tools.

CC @MNeisler to confirm the above

• ppelberg reassigned this task from DLynch to MNeisler.Sep 6 2022, 3:36 PM

This sounds okay to me considering that in the analysis that depends on this instrumentation (T298062) we'll be looking at edit completion rate which > – I do not think – would require us to look at anything more detailed than the first time someone entered text into the Reply or New Topic Tools.
CC @MNeisler to confirm the above

Yes, confirmed - "enters text" should just fire an event when the person first starts entering text rather than every single keystroke.

MNeisler reassigned this task from MNeisler to DLynch.Sep 9 2022, 6:43 PM

• ppelberg removed a parent task: T282638: [RELEASE TICKET] Offer Mobile Reply and New Discussion Tools at Partner Wikis.Sep 10 2022, 12:27 AM

• ppelberg moved this task from Code Review to QA on the Editing-team (Kanban Board) board.Sep 13 2022, 5:14 PM

• ppelberg added a project: Editing QA.

• ppelberg moved this task from Inbox to High Priority on the Editing QA board.

Change 835635 had a related patch set uploaded (by DLynch; author: DLynch):

[operations/mediawiki-config@master] MobileWebUIActions sample rate to 1 on testwiki

https://gerrit.wikimedia.org/r/835635

gerritbot added a project: Patch-For-Review.Sep 27 2022, 2:43 PM

Change 835635 merged by jenkins-bot:

[operations/mediawiki-config@master] MobileWebUIActions sample rate to 1 on testwiki

https://gerrit.wikimedia.org/r/835635

Mentioned in SAL (#wikimedia-operations) [2022-09-27T20:04:16Z] <samtar@deploy1002> Started scap: Backport for [[gerrit:835635|MobileWebUIActions sample rate to 1 on testwiki (T302108)]]

Mentioned in SAL (#wikimedia-operations) [2022-09-27T20:04:40Z] <samtar@deploy1002> samtar and kemayo: Backport for [[gerrit:835635|MobileWebUIActions sample rate to 1 on testwiki (T302108)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-09-27T20:10:03Z] <samtar@deploy1002> Finished scap: Backport for [[gerrit:835635|MobileWebUIActions sample rate to 1 on testwiki (T302108)]] (duration: 05m 46s)

Maintenance_bot removed a project: Patch-For-Review.Sep 27 2022, 8:30 PM

EAkinloose updated the task description. (Show Details)Sep 28 2022, 2:46 PM

Here are some of the logs I could catch. Thanks @DLynch

EAkinloose moved this task from QA to Ready for Sign Off on the Editing-team (Kanban Board) board.Sep 28 2022, 2:48 PM

Assigning over to me to do a quick QA of the aggregate data to confirm they are logging correctly.

• ppelberg moved this task from Ready for Sign Off to Blocked / Needs More Work on the Editing-team (Kanban Board) board.Oct 18 2022, 12:40 AM

MNeisler moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.Oct 26 2022, 2:32 PM

While QA'ing this data, I realized that sampling needs to be extended to additional wikis. In T309260, we adjusted sampling rates for this schema to 20% at the 9 partner wikis identified for the initial impact analysis. This was done to synchronize the sampling rate in the UIActions schemas and EditAttemptStep. We will also need to make sure sampling is turned on at this schema for the wikis identified for the DiscussionTools on Mobile AB Test (to be identified in T314950) prior to the AB test start.

Besides that, I've confirmed that all the new events added to track MobileFrontend's replying and new topic workflow are logging as expected in MobileWebUIActionTracking. See summary of check and some initial data below:

Summary of Checks

Confirmed that all 6 new events are logged and the frequency of each event type seems expected. The talkpage.add-topic is the most frequent followed by talkpage.focus-comment.

Frequency of MobileFrontEnd DT events (logged on 20 Oct 2022)

name	pct_events
talkpage.add-topic	52 %
talkpage.focus-comment	17.07 %
talkpage.input-comment	4.8 %
talkpage.input-topic	13.87 %
talkpage.publish-comment	4 %
talkpage.publish-topic	8.27 %

All these DT related events are associated with the event.action = click (versus init or show)
Events are logged on expected namespaces (47% on user talk pages and 42% on article talk pages)
Events are logged for both anon and registered users as expected.
An associated pageToken is logged with all events, which will be needed for any joins with EditAttemptStep data

In T302108#8347348, @MNeisler wrote:

While QA'ing this data, I realized that sampling needs to be extended to additional wikis. In T309260, we adjusted sampling rates for this schema to 20% at the 9 partner wikis identified for the initial impact analysis. This was done to synchronize the sampling rate in the UIActions schemas and EditAttemptStep. We will also need to make sure sampling is turned on at this schema for the wikis identified for the DiscussionTools on Mobile AB Test (to be identified in T314950) prior to the AB test start.

Great spot, @MNeisler.

Per what you and I talked about offline on 26 October 2022, the work to make the sampling rate adjustments you described above will happen in T321734.

Besides that, I've confirmed that all the new events added to track MobileFrontend's replying and new topic workflow are logging as expected in MobileWebUIActionTracking. See summary of check and some initial data below:

With the two quoted bits above in mind, I'm going to resolve this task.

Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptNov 2 2022, 5:48 PM

	F35538110: Screenshot 2022-09-28 at 15.39.56.png
	Sep 28 2022, 2:48 PM

	F35538108: Screenshot 2022-09-28 at 15.39.19.png
	Sep 28 2022, 2:48 PM

	F35538106: Screenshot 2022-09-28 at 15.37.44.png
	Sep 28 2022, 2:48 PM

	F35538103: Screenshot 2022-09-28 at 15.36.52.png
	Sep 28 2022, 2:48 PM

Ensure logging is in place to compare MobileFrontend and DiscussionTools new topic and new comment completion ratesClosed, ResolvedPublicActions