Page MenuHomePhabricator

Track edit button clicks in sticky header in the EditAttemptStep schema
Closed, ResolvedPublic

Description

This task involves the work of adding a new event to track when people click on the edit affordances ([[ ]], ✏️, and Add topic (T304187)) being introduced within the site-wide sticky header.

Requirements

  • The necessary event(s) are added to EditAttemptStep that enables us to track:
    • Edit sessions initiated via clicks on the edit affordances present within the site-wide sticky header
    • The editing interface people used the edit affordance within the site-wide sticky header to open
NOTE: As @MNeisler noted offline, the requirements above will likely require one new event [rather than three] considering we will be able to know the editing interface people are using a particular edit affordance within the sticky header to open by way of EditAttemptStep's existing editor_interface and integration fields.

Implementation Details

ScenarioData to be logged in EditAttemptStep
When someone opens the WikiEditor by clicking the [[ ]] button within the site-wide sticky headeraction = init, integration = page, init_type = page, init_mechanism = 'click-sticky-header', editor_interface = wikitext. See: T309364.
When someone opens the visual editor by clicking the ✏️ button within the site-wide sticky headeraction = init, integration = page, init_type = page, init_mechanism = 'click-sticky-header', editor_interface = visualeditor
When someone opens the New Topic Tool by clicking the Add topic button within the site-wide sticky headeraction = init, integration = discussiontools, init_type = section, init_mechanism = 'click-sticky-header'
When someone opens the legacy new topic workflow by clicking the Add topic button in the site-wide sticky headeraction = init, integration = page, init_type = section, init_mechanism = 'click-sticky-header'

Note: the details above are borrowed from T298634#7855485.

Background

As part of rolling out the sticky header feature with edit buttons, we want to make sure that edit button clicks are captured in the EditAttemptStep schema (see T298518).

Minimal test case

Acceptance criteria

  • Once edit buttons are included in the sticky header, edit clicks are logged in EditAttemptStep in a way that meets the ===Requirements above
  • Editing QA verifies the new event(s) added as part of this task are being emitted as expected by clients
  • @MNeisler to verify that aggregate data is logging to EditAttemptStep in the ways we expect it to

Event Timeline

@MNeisler - we've confirmed that we are able to track clicks within the sticky header T298518: [SPIKE] Tracking clicks to edit button within the sticky header. Do you know if we would need anything else to be able to track them across to the EditAttemptStep schema?

ppelberg subscribed.

@MNeisler - we've confirmed that we are able to track clicks within the sticky header T298518: [SPIKE] Tracking clicks to edit button within the sticky header. Do you know if we would need anything else to be able to track them across to the EditAttemptStep schema?

@ovasileva: per T298284#7783758, the Editing Team will be responsible for adding the new event(s) necessary to track clicks on the edit affordances within the sticky header.

I'm assigning this over to @MNeisler to propose name(s) for the new event(s) we'll need to add to meet the ===Requirements listed in the task description.

MNeisler triaged this task as Medium priority.Mar 17 2022, 1:30 AM
MNeisler moved this task from Triage to Current Quarter on the Product-Analytics board.
MNeisler moved this task from Doing to Needs Review on the Product-Analytics (Kanban) board.
MNeisler added a subscriber: DLynch.

Based on the current EditAttemptStep set-up, it seems this new event would fit most appropriately under the init_mechanism field in EditAttemptStep field.

Proposed new event name:
init_mechanism = 'click-sticky-header': Editing has been initiated from the sticky header

With this new event, we should be able to distinguish the different planned edit affordances in combination with the other fields as follows:

  • [[ ]] : action = init, integration = page, init_type = page, init_mechanism = 'click-sticky-header', editor_interface = wikitext
  • ✏️, : action = init, integration = page, init_type = page, init_mechanism = 'click-sticky-header', editor_interface = visualeditor
  • Add topic (New Topic Tool): action = init, integration = discussiontools, init_type = section, init_mechanism = 'click-sticky-header'
  • Add topic (Legacy Section = New): action = init, integration = page, init_type = section, init_mechanism = 'click-sticky-header'*

@DLynch - Any concerns with the above naming or implementation? Reassigning to you to review and confirm but let me know if you have any suggested changes.

Note: I considered also including within the init_type field but that would likely require two events (init_type = page-sticky-header and init_type = section-sticky-header) since you can access both full-page editing (using the edit button) and section editing (using the add topic button) within the Sticky Header. I'm ok with this though if that causes less issues in implementation.

This can be easily tracked for anything but WikiEditor -- it's logged server-side and wouldn't be distinguishable there from coming via existing methods, unless we add some otherwise-superfluous tracking parameters to the URLs.

This can be easily tracked for anything but WikiEditor -- it's logged server-side and wouldn't be distinguishable there from coming via existing methods, unless we add some otherwise-superfluous tracking parameters to the URLs.

In the long run, it will be valuable for us to be able to trace WikiEditor edit sessions that originate via people clicking on the new edit buttons ( [[ ]] and ✏️) within the sticky header seems worthwhile to me.

As such, I think it would be worthwhile for us to add the kind of tracking parameter to the URLs @DLynch referenced above.

Here is a ticket to implement the above: T309364.


Note with T309364 being filed, I think work on this ticket can begin.

ppelberg added a project: Goal.

Deployment Timing
Per today's planning meeting, the Editing Team intends to deploy the patch(es) that will resolve this task this Wednesday, 15 June via a backport window.

Change 805728 had a related patch set uploaded (by DLynch; author: DLynch):

[schemas/event/secondary@master] EditAttemptStep: add new values for init_mechanism

https://gerrit.wikimedia.org/r/805728

Change 805731 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/WikiEditor@master] Log if an edit attempt began from the sticky header

https://gerrit.wikimedia.org/r/805731

Change 805732 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@master] Log if an edit attempt began from the sticky header

https://gerrit.wikimedia.org/r/805732

Change 805729 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/WikimediaEvents@master] Bump EditAttemptStep version to 1.3.0

https://gerrit.wikimedia.org/r/805729

Change 805728 merged by jenkins-bot:

[schemas/event/secondary@master] EditAttemptStep: add new values for init_mechanism

https://gerrit.wikimedia.org/r/805728

Change 805729 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Bump EditAttemptStep version to 1.3.0

https://gerrit.wikimedia.org/r/805729

Jdlrobson subscribed.

Looks like the web team is needed only on T309364 which we'll get to today. Let me know if you need any input on this task.

Change 805731 merged by jenkins-bot:

[mediawiki/extensions/WikiEditor@master] Log if an edit attempt began from the sticky header

https://gerrit.wikimedia.org/r/805731

Change 805732 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Log if an edit attempt began from the sticky header

https://gerrit.wikimedia.org/r/805732

Ryasmeen subscribed.

The first two events are being emitted as expected:

Screen Shot 2022-06-27 at 4.22.19 PM.png (686×1 px, 181 KB)

Screen Shot 2022-06-27 at 4.22.57 PM.png (676×1 px, 170 KB)

Waiting for the "Add Topic" button to be available on the sticky header to test the remaining scenarios.

@Ryasmeen - for this A/B test we'll only be looking at the addition of the edit button, not sure if there's another task to test for the add topic button

@ovasileva This seems to be good enough for you all to start the A/B testing with the edit button.

I know this task is including more details than just the edit button. I'll leave it for @ppelberg to decide what would be next. and @Ryasmeen can continue testing once the other patch is merged. I think it is https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/789614

Next step

  • @ppelberg to file task for testing the Add topic instrumentation is working as expected prior to T304372 being deployed.

I checked the data logged in EditAttemptStep and confirmed that events where init_mechanism == 'click-sticky-header' appears to be logging as expected. See a summary of checks and observations below:

  • We start logging these events on 6 July when the AB test was deployed to the participating wikis.
  • We are logging clicks to open both the WikiEditor and VisualEditor buttons in the Sticky Header. The majority (about 71% ) of the clicks have been to the VisualEditor button (See breakdown in table below):
  • All of these new events occur for init_type = 'page' and integration = 'page', which is expected as the add topic button has not yet been added to the sticky header.
  • There have been no sticky header edit button clicks on talk pages. The majority of these edit attempts have occurred on the main article namespace.
  • These events are only logged at participating wikis where the AB Test was deployed.

Sticky Header Edit Button Events Logged from 6 July 2022 through - 18 July 2022

editor_interfaceinit_typeintegrationn_clicksn_users
visualeditorpagepage273177
wikitextpagepage7964
wikitext-2017pagepage98

cc @DLynch @ppelberg

With @MNeisler confirming in T298634#8085308 that we are tracking editing-related sticky header events as expected and @ovasileva confirming offline that the Web Team is going to do the QA necessary to ensure the A/B test buckets are balanced, I'm going to resolve this task.