Page MenuHomePhabricator

[SPIKE] Explore instrumentation for sticky header work
Open, HighPublic

Description

Background

The goal of the sticky header is to make functionality usually found at the top of the page more easily accessible from everywhere within the page. We would like to measure this through getting a comparison of the clickthrough of the links at the top of the page vs the sticky header, as well as by measuring whether it decreases the need to scroll to the top of the page

Questions

  1. Does the sticky header decrease the need to stroll to the top of the page?
  2. Is the functionality added to the sticky header used and how frequently?
    1. What is the clickthrough rate (per pageview or per session) of each item on the sticky header (Note: Maybe overall number of clicks can answer the question)
    2. What is the ratio of clicks of sticky header items to the corresponding items at the top of the page
    3. Is there an overall increase in clicks to the functionality available in the sticky header (i.e. the sum of the sticky header functionality and the corresponding items at the top of the page)
    4. What is the effect of the sticky header introduction on edit rates for editors of varying experience levels (by edit bucket)

Rough plan

  • Add new events to https://meta.wikimedia.org/wiki/Schema:DesktopWebUIActionsTracking that correspond to links within the sticky header
  • Create a new schema that instruments scrolls to the top of the page (potentially by measuring the viewport and triggering an event beyond a certain threshold)
  • Perform an A/B test with the test group seeing the sticky header and the control group seeing the current experience

Next steps in data process (bolding indicates step we are currently on):

  • Data analyst creates a list of events that need to be tracked to calculate each metric
  • Engineers review the event list and comment on whether we’re already tracking with existing instruments and if new instrumentation is required. New tickets created for any new instrumentation needed.
  • Data analyst works with engineers to figure out what the new instrumentation should look like and where to store the events.
  • Data analyst documents all of the above in the instrumentation spec
  • PM signs off on the plan

Event Timeline

ovasileva triaged this task as Medium priority.Jul 29 2021, 8:49 PM
ovasileva added a subscriber: phuedx.
Jdlrobson added subscribers: MNeisler, Jdlrobson.

@ovasileva can I clarify the purpose of this spike? Is the question really "How do we measure the impact of sticky header work?"

There's not much to spike around scrolling. Using the scroll API we can track when the user scrolls and model behaviours such as scrolled down, scrolled up, and even scroll speed. There are issues around privacy here and performance that we will need to consider, but it seems useful to first decide what we need to measure.

we would like to identify how to measure when a user scrolls

We can measure when a user scrolls no problem. No need for a spike on this.

in order to load vue at the correct time.

I think this is a separate concern, covered in the other spike T287529

In addition, we would like to be able to measure if there's any decrease in visits to the top of the page within a single pageview or a decrease in overall fast scrolling per page

Performance concerns aside, we can track when a user is at the top of a page either on page load, or later in their browsing experience. We could also track their speed in terms of pixels per second.
I suggest we follow @MNeisler's process and begin by generating the ''Instrumentation Spec Template'' ( https://docs.google.com/spreadsheets/d/1HxvmPJSPc3qNzVYopdZa6YI5A2DcDHHs2xd0D1svuuk/edit#gid=0 ) for this project. with Jennifer. It feels premature to look at implementation details without this. Perhaps we should setup a meeting with Jennifer?

ovasileva renamed this task from [SPIKE] Explore scrolling for sticky header work to [SPIKE] Explore instrumentation for sticky header work.Mon, Aug 30, 8:50 PM
ovasileva updated the task description. (Show Details)
ovasileva added a subscriber: jwang.

@jwang - Assigning to you as per our conversation for generating the list of events

ovasileva raised the priority of this task from Medium to High.Thu, Sep 2, 4:46 PM