Page MenuHomePhabricator

Investigate sample weighting of search metrics
Open, NormalPublic

Description

There’s a concern that the sampling applied across languages/projects/platforms is not weighted properly. Check on that and fix and/or document as needed.

For example, the recent fix to the iOS bug around May 11 seems to have had an unexpectedly large effect on User Engagement (https://discovery.wmflabs.org/metrics/#kpi_augmented_clickthroughs). There seems to be something wrong in either the weighting of the sampling across platforms, or some people’s understanding of what it measures. (E.g., you can have an unweighted mean of rates to get a sense of “average rate”, but that’s not what most people expect from a unified rate measure.)

Event Timeline

EBjune created this task.Jun 13 2018, 3:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 13 2018, 3:39 PM
TJones added a subscriber: TJones.Jun 14 2018, 5:48 PM
chelsyx moved this task from Triage to Backlog on the Product-Analytics board.
chelsyx added a subscriber: chelsyx.
Vvjjkkii renamed this task from Investigate sample weighting of search metrics to f3aaaaaaaa.Jul 1 2018, 1:05 AM
Vvjjkkii triaged this task as High priority.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot renamed this task from f3aaaaaaaa to Investigate sample weighting of search metrics.Jul 2 2018, 1:51 PM
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot added a subscriber: Aklapper.
MBinder_WMF triaged this task as Normal priority.Aug 2 2018, 8:20 PM