Page MenuHomePhabricator

Ensure logging is in place to compare MobileFrontend and DiscussionTools new topic and new comment completion rates
Closed, ResolvedPublic

Description

Part of the work we will be doing to evaluate the impact of the collection of improvements the Editing Team will be making to mobile talk pages (T278588) will involve conducting two impact analyses (T298058 + T298065).

In order to conduct these analyses, we will need to ensure sufficient instrumentation is in place to compare the DT-enhanced version of mobile web talk pages to the existing, Read as wiki page and MobileFrontend talk overlay, versions of mobile web talk pages.

Requirements

  • The MobileWebUIActions schema is logging the following events within the MobileFrontend's replying workflow:
    • When a person taps "into" the reply input field
    • When a person enters a character into the reply input field
    • When a person publishes a comment they drafted
  • The MobileWebUIActions schema is logging the following events within the MobileFrontend's new topic workflow:
    • When a person initiates the workflow by way of tapping the Add topic button
    • When a person enters text into either of the new topic workflow's fields (Subject or Make a suggestion or voice a concern)
    • When a person publishes a topic they drafted
NOTE: the answers to ===Open Questions #1 and/or #2 below may mean no additional work is required to achieve some, if not all, of the requirements listed above.

Testing

actionuiactions eventnewly added?Client-side logging verified?
When a person taps "into" the reply input fieldtalkpage.focus-commentYesYes
When a person enters a character into the reply input fieldtalkpage.input-commentYesYes
When a person publishes a comment they draftedtalkpage.publish-commentYesYes
When a person initiates the workflow by way of tapping the Add topic buttontalkpage.add-topicYes
When a person enters text into either of the new topic workflow's fieldstalkpage.input-topicYesYes
When a person publishes a topic they draftedtalkpage.publish-topicYesYes

Open Questions

  • 1. What – if any – events within the MobileFrontend's new topic workflow are currently being logged?
  • 2. What – if any – events within the MobileFrontend's replying workflow

are currently being logged?

Done

  • Answers to all === Open Questions are documented
  • All ===Requirements are met
  • @Ryasmeen / @EAkinloose verify clients are emitting the newly-added events listed in the ===Testing section above
  • @MNeisler verifies the newly-added events are being logged in the DB as expected

i. @DLynch surfaced this idea during the Editing Team's 16 February 2022 team meeting (see Editing Scratch Archive for notes).

Related Objects

Event Timeline

I'll note that a maybe-important aspect that'd be hard to compare would be "edits that aren't even attempted" in the current mobile overlay -- because with DT we're adding a new ability to reply to nested comments, and so there's presumably people who would have previously bounced off the page without leaving any trace on EditAttemptStep who will now instead make an edit.

Per what @DLynch, @MNeisler, and I talked about offline last week (18 May 2022):

  1. We DECIDED that rather than adding EditAttemptStep schema events to the existing MobileFrontend talk page overlay, we will instead verify/ensure that the MobileFrontend includes sufficient logging for us to compare when people start and publish an edit using its existing new topic and commenting workflows
  2. We DECIDED that for the purpose of comparing the edit completion rates between the Reply Tool and the existing MobileFrontend commenting experience, we will consider the denominator in the edit completion rate to be the firstChange event within the Reply Tool/EditAttemptStep schema.
  3. In line with "1." and "2." we're going to reframe this task as being about investigating what - if any – instrumentation needs to be added to the existing MobileFrontend new topic and commenting experiences in order to compare the rates at which people complete the comments and new topics they start with the MobileFrontend talk tools and the mobile Discussion Tools.
ppelberg renamed this task from [SPIKE] Investigate the value and work involved with adding EditAttemptStep to the mobile talk overlay to Ensure logging is in place to compare MobileFrontend and DiscussionTools new topic and new comment completion rates.May 26 2022, 3:07 AM
ppelberg updated the task description. (Show Details)

Change 827530 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/WikimediaEvents@master] Let uiactions logging be triggered via mw.track

https://gerrit.wikimedia.org/r/827530

Change 827531 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/MobileFrontend@master] Track various talk overlay events via UIActions

https://gerrit.wikimedia.org/r/827531

actionuiactions eventnewly added?
When a person taps "into" the reply input fieldtalkpage.focus-commentY
When a person enters a character into the reply input fieldtalkpage.input-commentY
When a person publishes a comment they draftedtalkpage.publish-commentY
When a person initiates the workflow by way of tapping the Add topic buttontalkpage.add-topic
When a person enters text into either of the new topic workflow's fieldstalkpage.input-topicY
When a person publishes a topic they draftedtalkpage.publish-topicY

Change 827530 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Let uiactions logging be triggered via mw.track

https://gerrit.wikimedia.org/r/827530

Change 827531 merged by jenkins-bot:

[mediawiki/extensions/MobileFrontend@master] Track various talk overlay events via UIActions

https://gerrit.wikimedia.org/r/827531

ppelberg updated the task description. (Show Details)
ppelberg updated the task description. (Show Details)
ppelberg added subscribers: EAkinloose, Ryasmeen.

It occurs to me to clarify: I interpreted both of the "enters text" requirements as requiring that an event is fired only the first time this happens, rather than for every single keystroke.

It occurs to me to clarify: I interpreted both of the "enters text" requirements as requiring that an event is fired only the first time this happens, rather than for every single keystroke.

This sounds okay to me considering that in the analysis that depends on this instrumentation (T298062) we'll be looking at edit completion rate which – I do not think – would require us to look at anything more detailed than the first time someone entered text into the Reply or New Topic Tools.

CC @MNeisler to confirm the above

This sounds okay to me considering that in the analysis that depends on this instrumentation (T298062) we'll be looking at edit completion rate which > – I do not think – would require us to look at anything more detailed than the first time someone entered text into the Reply or New Topic Tools.
CC @MNeisler to confirm the above

Yes, confirmed - "enters text" should just fire an event when the person first starts entering text rather than every single keystroke.

ppelberg added a project: Editing QA.
ppelberg moved this task from Inbox to High Priority on the Editing QA board.

Change 835635 had a related patch set uploaded (by DLynch; author: DLynch):

[operations/mediawiki-config@master] MobileWebUIActions sample rate to 1 on testwiki

https://gerrit.wikimedia.org/r/835635

Change 835635 merged by jenkins-bot:

[operations/mediawiki-config@master] MobileWebUIActions sample rate to 1 on testwiki

https://gerrit.wikimedia.org/r/835635

Mentioned in SAL (#wikimedia-operations) [2022-09-27T20:04:16Z] <samtar@deploy1002> Started scap: Backport for [[gerrit:835635|MobileWebUIActions sample rate to 1 on testwiki (T302108)]]

Mentioned in SAL (#wikimedia-operations) [2022-09-27T20:04:40Z] <samtar@deploy1002> samtar and kemayo: Backport for [[gerrit:835635|MobileWebUIActions sample rate to 1 on testwiki (T302108)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-09-27T20:10:03Z] <samtar@deploy1002> Finished scap: Backport for [[gerrit:835635|MobileWebUIActions sample rate to 1 on testwiki (T302108)]] (duration: 05m 46s)

Here are some of the logs I could catch. Thanks @DLynch

Screenshot 2022-09-28 at 15.36.52.png (149×1 px, 54 KB)

Screenshot 2022-09-28 at 15.37.44.png (147×1 px, 55 KB)

Screenshot 2022-09-28 at 15.39.19.png (150×1 px, 58 KB)

Screenshot 2022-09-28 at 15.39.56.png (146×1 px, 55 KB)

MNeisler triaged this task as Medium priority.

Assigning over to me to do a quick QA of the aggregate data to confirm they are logging correctly.

While QA'ing this data, I realized that sampling needs to be extended to additional wikis. In T309260, we adjusted sampling rates for this schema to 20% at the 9 partner wikis identified for the initial impact analysis. This was done to synchronize the sampling rate in the UIActions schemas and EditAttemptStep. We will also need to make sure sampling is turned on at this schema for the wikis identified for the DiscussionTools on Mobile AB Test (to be identified in T314950) prior to the AB test start.

Besides that, I've confirmed that all the new events added to track MobileFrontend's replying and new topic workflow are logging as expected in MobileWebUIActionTracking. See summary of check and some initial data below:

Summary of Checks

  • Confirmed that all 6 new events are logged and the frequency of each event type seems expected. The talkpage.add-topic is the most frequent followed by talkpage.focus-comment.

Frequency of MobileFrontEnd DT events (logged on 20 Oct 2022)

namepct_events
talkpage.add-topic52 %
talkpage.focus-comment17.07 %
talkpage.input-comment4.8 %
talkpage.input-topic13.87 %
talkpage.publish-comment4 %
talkpage.publish-topic8.27 %
  • All these DT related events are associated with the event.action = click (versus init or show)
  • Events are logged on expected namespaces (47% on user talk pages and 42% on article talk pages)
  • Events are logged for both anon and registered users as expected.
  • An associated pageToken is logged with all events, which will be needed for any joins with EditAttemptStep data

While QA'ing this data, I realized that sampling needs to be extended to additional wikis. In T309260, we adjusted sampling rates for this schema to 20% at the 9 partner wikis identified for the initial impact analysis. This was done to synchronize the sampling rate in the UIActions schemas and EditAttemptStep. We will also need to make sure sampling is turned on at this schema for the wikis identified for the DiscussionTools on Mobile AB Test (to be identified in T314950) prior to the AB test start.

Great spot, @MNeisler.

Per what you and I talked about offline on 26 October 2022, the work to make the sampling rate adjustments you described above will happen in T321734.

Besides that, I've confirmed that all the new events added to track MobileFrontend's replying and new topic workflow are logging as expected in MobileWebUIActionTracking. See summary of check and some initial data below:

With the two quoted bits above in mind, I'm going to resolve this task.