Page MenuHomePhabricator

Decide on better wording for "users were 1.07 times more likely to do X"
Closed, ResolvedPublic

Description

In https://www.mediawiki.org/wiki/Topic:T7cyg64hzu6aop69 someone pointed out that the sentence "We found that the test group had a 3.64% higher probability of interacting with the page and was 1.07 times more likely to engage with the page than the control group." can be confusing to people who don't specialize in math.

People not well-versed in math will have a hard time translating the 1.07 into a recognition that engagement increased by 75. I proposed replacing "was 1.07 times more likely to" with "was 7% more likely to", but @mpopov expressed concerns that the percentage wording would mislead people into thinking it was a probability.

The goal of this task is to find wording that is:

  • Correct
  • Accessible to people who are not specialists in math
  • Unlikely to lead to misinterpretations

Perhaps this has already been "solved" by folks in the Analytics team?

Event Timeline

Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptJul 12 2016, 11:34 PM
debt triaged this task as Normal priority.Jul 13 2016, 6:55 PM
debt added a subscriber: debt.

moving to the sprint board as this topic is undergoing discussion now.

We are going to wait until our new analyst starts working because they will probably have very valuable insights on this topic.

Deskana changed the task status from Open to Stalled.Jul 23 2016, 6:12 PM
Deskana added a subscriber: Deskana.

We are going to wait until our new analyst starts working because they will probably have very valuable insights on this topic.

Marking as stalled based on this.

@chelsyx, @debt, and I talked about this and decided to try referring to things in terms of "out of a hundred" e.g.

if P(Clickthrough in Control Group) = 0.7, P(Clickthrough in Test Group) = 0.75,

then "in the control group without the new feature, we can expect 70 out of 100 sessions to click on a search result, while we can expect 75 out of 100 sessions with the new feature to click on a search result"

this is also credible interval-friendly: "we can expect 2-9 more sessions to click on a search result when they have the new feature"

@debt: it's done but don't resolve this yet until we end up using this phrasing in a report and people find it better (more accessible without losing accuracy)

I look forward to seeing this in context, but based on that fragment it looks awesome so far.

debt added a comment.Sep 8 2016, 11:07 PM

Great! :)

debt closed this task as Resolved.Sep 9 2016, 6:51 PM
debt claimed this task.
Deskana reassigned this task from debt to mpopov.Sep 9 2016, 9:51 PM

Updating assignment for clarity.