Page MenuHomePhabricator

Spike: How do we measure success of link previews on mobile?
Closed, ResolvedPublic

Description

We need to be able to show that we have met our numerical targets. How would we do this? Person carrying out spike should talk to Jon Katz and analytics team members.

Outcome: A wiki page + comment on this bug report summarising how we are going to show success.
Duration: 4hrs
Acceptance criteria:
We need to know

  • What will we measure / What does success look like? Increase in number of clicks?
  • How much are we looking to increase the chosen metric? What is the ratio of click increases we are looking to get to?
  • What will we use to show success? Will we use SQL queries or a graph?
NOTE: this will be different on desktop

Event Timeline

Jdlrobson assigned this task to dr0ptp4kt.
Jdlrobson raised the priority of this task from to Normal.
Jdlrobson updated the task description. (Show Details)

Adam had some links / information on this.

Jdlrobson renamed this task from How do we measure link previews? to Spike: How do we measure link previews?.Sep 24 2015, 5:42 PM
Jdlrobson set Security to None.

@Jdlrobson, take a look at what Dmitry measured for Android link preview. Basically, the key metric was clicks on links--do those go up?

But they measured a number of other things like - time spent on link preview before continuing on or dismissing it..etc.

Here is the schema: https://meta.wikimedia.org/wiki/Schema:MobileWikiAppLinkPreview

Here are beta results (ignore the production for now): https://docs.google.com/spreadsheets/d/1FOSTa2iplla6LvULlqSVG2pkPc88UkbxxXKsBmEwD_0/edit#gid=2098423693

Jdlrobson renamed this task from Spike: How do we measure link previews? to Spike: How do we measure success of link previews on mobile?.Sep 24 2015, 5:43 PM
Jdlrobson updated the task description. (Show Details)
Jdlrobson updated the task description. (Show Details)
JKatzWMF updated the task description. (Show Details)Sep 24 2015, 5:44 PM
Jdlrobson updated the task description. (Show Details)
Jdlrobson updated the task description. (Show Details)
Jdlrobson removed dr0ptp4kt as the assignee of this task.Sep 24 2015, 5:50 PM
Jdlrobson updated the task description. (Show Details)
Jdlrobson removed a project: MobileFrontend.

@JKatzWMF would it be useful to do this as part of this sprint?

@Jdlrobson, the spreadsheet @JKatzWMF provided covers analyses helpful for this sort of thing. You'll want to check with @Tbayer, and @Dbrant about what sorts of queries you might need.

Do note on the web sendBeacon can determine whether an event is actually sent to the server if, say, the user tapped through from a preview to the full article. sendBeacon is not universally supported.

For meeting the Q1 primary goal as expressed at https://www.mediawiki.org/w/index.php?title=Wikimedia_Engineering/2015-16_Q1_Goals&oldid=1902915#Reading , "10% increase in Android app user engagement as defined: (Full Pageviews + TextExtracts retrievals with link preview feature) >= 1.1 * (Pageviews prior to link preview)". Looking at this on an aggregate basis is fine, although looking at it on a per-user average (and distribution curve) basis is more telling and reduces artifacts like seasonality.

Gotchas for web:

  • not all UAs support sendBeacon, so although some browsers get lucky with events arriving during a page transition, it's not a sure thing
  • when doing per user averages, you need a means of counting the number of unique users using some heuristic if doing per-user averages or distribution curves

It may be easier to simply tag X-Analytics at the content service endpoint so that it reflects the case when the retrieval is a link preview. If you do this, it will be easier to query aggregates at Hive without having to worry about sendBeacon. You still have to worry about how to count the number of user sessions, though, for per-user averages or distribution curves.

@dr0ptp4kt spike implementor should thus contact the people you've poked. Is there anything missing from the acceptance criteria?

Jdlrobson updated the task description. (Show Details)Sep 28 2015, 10:50 PM
Jdlrobson updated the task description. (Show Details)

@Jdlrobson, which acceptance criteria?

JKatzWMF closed this task as Resolved.Nov 18 2015, 1:08 AM

I think we have enough here for the event logging in T113634