Page MenuHomePhabricator

Instrument the revised "Add topic" button treatment
Closed, ResolvedPublic

Description

T267444, and eventually T326694, will revise where and how the Add Topic button appears on talk pages.

This task involves the work of instrumenting said Add Topic button so that we can evaluate who is engaging with it and what they are doing after doing so. More details below.

Requirements

The Add Topic is instrumented such that we can know:

  • The account state of the people who click it (logged in / out)
  • The experience level (edit count) of the people who click it
  • The skin version being shown when the button is clicked (i.e. Vector 2010 or Vector 2022)
  • The editing interface that people see once they click the Add Topic button
  • The editing workflow that people use once they click the Add Topic button (i.e. New Topic Tool or section=new workflow)
  • Where within the new topic publishing flow people "get to." Read: do they abandon the workflow before the editing interface loads? Do they abandon the workflow after entering some content into the editing interface? Do they successfully publish an edit? etc.
NOTE: Ideally, this "new" button will be instrumented in the same way(s) the existing Add Topic button. This includes the same click-logging in DesktopWebUIActions and the same add topic workflow instrumentation within EditAttemptStep

Open questions

  • 1. Considering we're implementing this new Add Topic button in Vector (2010) and Vector (2022) separately, does the instrumentation of both need to happen separately as well? Can it happen all in this one task? Something else?
    • The instrumentation for the Add Topic button that will appear in Vector (2010) and Vector (2022) will, ideally, happen in this task assuming the The skin version being shown when the button is clicked (i.e. Vector 2010 or Vector 2022) requirement can be met.

Event Timeline

ppelberg created this task.
ppelberg moved this task from Backlog to Triaged on the DiscussionTools board.
ppelberg added a subscriber: Esanders.

@MNeisler: before we begin work on implementation, can you please review the requirements and make sure what I've written will provide us everything we need to report on the metrics that we've currently specified in T302358?

I've reviewed the requirements and made some additions in the task description. The requirements should now cover everything needed to report on the metrics specified in T302358.

Summary of updates:

  • I added a requirement to make sure that we can track the skin version being shown at the time of the click on the add topic button. Given that this new button is being implemented in both Vector 2010 and Vector 2022, this will ensure that we can decipher between the two buttons if needed (assuming that they are instrumented the same way). Note: DesktopWebUIActionsTracking includes a skin and skinVersion field which can be used to obtain this data.
  • In addition to instrumentation in DesktopWebUIActionsTracking, we will be using EditAttemptStep to track edit workflow after a user clicks on the button. This will be used to calculate edit completion rate in T302358. This will require us making sure that the sampling rates between the two schemas are synchronized at the partner wikis.
  • In EditAttemptStep, we should be able to track if the person is using the New Topic Tool or the legacy section=new workflow. This should be available in EditAttemptStep using the existing integration, init_type, and init_mechanism fields but adding so we have a complete requirement list.

All of these updates are reflected in the ==Requirements== section of the task description.

MNeisler updated the task description. (Show Details)
ppelberg raised the priority of this task from Medium to High.
ppelberg updated the task description. (Show Details)
ppelberg added a subscriber: MNeisler.

Change 895362 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/skins/Vector@master] Add logging to the add topic button when it's promoted to the header

https://gerrit.wikimedia.org/r/895362

As I understand the requirements, all that I need to do here is add the DesktopWebUIActionsTracking to the "add topic" button when it's promoted to the header. That patch logs it with the name "addsection-header". (As opposed to "ca-addsection" for the unpromoted version of the button, and "addsection-sticky-header" for the version in the sticky header.)

Change 895362 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Add logging to the add topic button when it's promoted to the header

https://gerrit.wikimedia.org/r/895362

DLynch moved this task from Code Review to QA on the Editing-team (Kanban Board) board.
DLynch added a project: Editing QA.

I checked events recorded in DesktopWebUIActionsTracking since this patch was deployed and confirmed events to the revised topic button (event.name = addsection-header) currently appear to be logging as expected.

Summary of click events to the revised add topic button promoted in the header (17 March 2023 through 20 March 2023):

  • There have been 26 total click events since instrumentation was deployed. See table showing number of clicks to different topic buttons below:
add topic button typenumber of clicks
addsection-header26
addsection-sticky-header1003
ca-addsection25153
  • This includes 2 click events on hu.wiki and 24 click events on cs.wiki (the two partner wikis where this was enabled in T331313 )
  • We start logging these events on March 17th
  • All events occurred on Vector 2022 skin version as expected
  • 18 click events by logged-in users and 8 click events for logged-out users.
  • Clicks events occurred on Visual Editor (23 events) or Wikitext-2017 (9 events). No wikitext events.
  • All currently logged as using the New Topic tool (integration = discussiontools)

cc @DLynch @ppelberg

DesktopWebUIActions, EditAttemptStep, and event metrics logged

Screenshot 2023-03-26 at 05.41.17.png (1×3 px, 696 KB)

Screenshot 2023-03-26 at 06.01.53.png (932×3 px, 718 KB)

Screenshot 2023-03-26 at 06.01.53.png (932×3 px, 694 KB)

Screenshot 2023-03-26 at 05.48.40.png (1×3 px, 868 KB)