Page MenuHomePhabricator

[Spike ??hrs] Sticky header instrumentation
Open, Stalled, LowPublic

Description

Precursor

Let's get a draft schema set before working on this. It will help guide the spike and find any problems with it.

Background

We would like to run an A/B test on the sticky headers navigation feature to determine whether users are navigating less while reading. We will be looking at the amount of scrolling up (and down) the page with the hypothesis that users in the test group will scroll up less per page.

We would like to know the following:

Before building out T197718 we want to determine how to instrument and test this feature.

@ovasileva @Tbayer @alexhollender @phuedx @Jdlrobson met to discuss options. We'll update this card with a summary of that conversation and associated spikes we'll need to carry out to do so.

We do not intend to run multiple A/B tests simultaneously.

Acceptance criteria

  • Determine the best way to instrument the feature so that we can collect the data and answer the questions outlined above
  • Write a task with the proposed implementation

Questions to answer

  • Which bits are simple
  • Which bits would be complicated/give rise to concern.
  • Any problems with the draft schema?
  • The height is variable depending on what is open and closed.

Notes

  • scroll event is triggered by user interactions, internal links (e.g. #Heading) and opening overlays (notably clicking images opens the media viewer overlay which triggers a scroll to top). @phuedx pointed out that looking at event.currentTarget.srcElement will allow us to distinguish scrollbar-triggered (srcElement === document) and UI element triggered scrolls.
  • We need to decide whether to send multiple events per page or follow an approach similar to https://meta.wikimedia.org/wiki/Schema:MobileWikiAppPageScroll

Event Timeline

Jdlrobson triaged this task as Normal priority.Jul 9 2018, 9:11 PM
Jdlrobson created this task.
Jdlrobson moved this task from Incoming to Needs Prioritization on the Readers-Web-Backlog board.
Jdlrobson updated the task description. (Show Details)
phuedx updated the task description. (Show Details)Jul 10 2018, 10:44 AM
pmiazga updated the task description. (Show Details)Jul 10 2018, 5:05 PM

@Aklapper - it seems I don't have the permissions to edit this task. Is there a way to check what is causing that?

Nuria moved this task from Incoming to Radar on the Analytics board.Jul 12 2018, 4:46 PM
Tbayer changed the edit policy from "Custom Policy" to "All Users".Jul 13 2018, 4:39 PM
ovasileva updated the task description. (Show Details)Jul 13 2018, 6:00 PM
ovasileva renamed this task from Sticky header instrumentation to [SPIKE] Sticky header instrumentation.Jul 17 2018, 3:51 PM
ovasileva renamed this task from [SPIKE] Sticky header instrumentation to [Spike] Sticky header instrumentation.
ovasileva updated the task description. (Show Details)
ovasileva updated the task description. (Show Details)Jul 17 2018, 3:53 PM

We need to decide whether to send multiple events per page or follow an approach similar to https://meta.wikimedia.org/wiki/Schema:MobileWikiAppPageScroll

Is this correct? I thought we'd decided to send one event per page.

ovasileva updated the task description. (Show Details)Jul 17 2018, 4:54 PM
ovasileva renamed this task from [Spike] Sticky header instrumentation to [Spike 8hrs] Sticky header instrumentation.Jul 17 2018, 4:58 PM
ovasileva moved this task from Needs Prioritization to Upcoming on the Readers-Web-Backlog board.

Discussed in standup and noted that the next way forward might be to create a draft of the schema in parallel to this task.

^ If creating a draft schema helps the investigator to think about the implementation, then they should feel free to create one!

number of sections opened/closed per page (we could potentially use the section usage schema: https://meta.wikimedia.org/wiki/Schema:MobileWebSectionUsage)

Note, we should also consider the fact that Special:MobileOptions allows users to expand all sections by default, a state which will impact instrumentation.

Jdlrobson added subscribers: nray, pmiazga, Stephen, jan.

Since there is no "points 0" on this task it means we haven't talked about it. We did an async estimation and estimations are all over the place.

@jan: 8, @pmiazga: 8, @Stephen: 5, @Jdlrobson: 13, @nray: needs analysis:banana:

Let's talk about this today and work out how to make this clearer.

Jdlrobson renamed this task from [Spike 8hrs] Sticky header instrumentation to [Spike ??hrs] Sticky header instrumentation.Aug 14 2018, 4:17 PM
Jdlrobson updated the task description. (Show Details)

Since there is no "points 0" on this task it means we haven't talked about it. We did an async estimation and estimations are all over the place.
@jan: 8, @pmiazga: 8, @Stephen: 5, @Jdlrobson: 13, @nray: needs analysis:banana:
Let's talk about this today and work out how to make this clearer.

I'm a bit confused. Why are we pointing this? In grooming we decided it was a spike (so hours, not points) and gave a rough estimate for 8 hours. I'm not arguing against looking at it again, just confused as to the process here.

Jdlrobson added a comment.EditedAug 17 2018, 2:55 PM

We tried to point the '''hours'''. We (devs+tilman) then chatted about the task in the grooming meeting and decided this was not ready to be worked on. A precursor has been added to the task which we believe is important before agreeing to a timebox and taking on this spike.

For clarity: we have been adding 0 points to the field of a spike task as a signal that we have talked about the task, agreed on the hours and said it is ready to be worked on.

This comment was removed by Jdlrobson.

For clarity: we have been adding 0 points to the field of a spike task as a signal that we have talked about the task, agreed on the hours and said it is ready to be worked on.

Got it! I think I was missing this part - did we decide on doing this at some point? Have I been missing it all along? (i.e. I haven't done this to any of the tasks I've added hours to - doesn't adding hours show that we have discussed it as a team as well?)

Restricted Application added a project: Product-Analytics. · View Herald TranscriptAug 21 2018, 4:57 PM

Got it! I think I was missing this part - did we decide on doing this at some point? H

It's a new thing that's been happening given we were getting confused about whether spikes had been discussed or not. While adding hours is one way, we've historically created tasks where one-two person has suggested/decided upon hours and these hours have changed and it's hard to distinguish. I've documented this here:
https://www.mediawiki.org/wiki/Reading/Web/Team/Story_estimation#Spikes_(0)
Comments are probably the most useful signal here (the where, the who, the what, the why) and I'll commit to doing better at that when updating points/hours in future.

Jdlrobson added a comment.EditedAug 23 2018, 12:25 AM

A few questions (no urgency right now)

number of sections opened/closed per page

@Tbayer I made a few edits to https://meta.m.wikimedia.org/wiki/Schema:MobileWebSectionUsage to update it with current code.

With it, if we were to send multiple events per page we would be able to count scrollIntoView events and close events and I think answer this question using the code we used before.

total number of pixels scrolled up and down per page

It feels like https://meta.m.wikimedia.org/wiki/Schema:ReadingDepth tells us how long a user spends on the page (visible time and total time spent on page). Would the plan be to add pageHeight,scrollFluxDown,scrollFluxUp,percentageViewed (or something along those lines) to that schema?

I think this is the tricky one we would want to investigate in this spike. Specifically how those fields get calculated.

nettrom_WMF moved this task from Triage to Doing on the Product-Analytics board.
Aklapper removed Tbayer as the assignee of this task.Apr 22 2019, 11:47 AM

Resetting task assignee as the user is not active here anymore.

ovasileva changed the task status from Open to Stalled.May 3 2019, 4:58 PM
ovasileva lowered the priority of this task from Normal to Low.

dropping as low until the project is scheduled