Page MenuHomePhabricator

Oversampling changes edit completion rate
Open, LowPublic

Description

Context

In T221197#5319337, we learned oversampling changes the edit completion rate between the two test groups: default-visual and default-source.

We do not currently know why this is.

Implications

If our randomization is not working properly (read: it is not truly random), the accuracy/legitimacy of all of our other of all of our test metrics is put into question.

"Done"

  • We have a reasonable explanation as to why oversampling changes the edit completion rate between the two test groups: default-visual and default-source.

Event Timeline

Neil_P._Quinn_WMF triaged this task as High priority.Jul 30 2019, 1:20 PM
Neil_P._Quinn_WMF moved this task from Triage to Doing on the Product-Analytics board.
Neil_P._Quinn_WMF lowered the priority of this task from High to Normal.Aug 20 2019, 6:26 PM
kzimmerman lowered the priority of this task from Normal to Low.Aug 27 2019, 6:32 PM
kzimmerman moved this task from Doing to Backlog on the Product-Analytics board.
kzimmerman added a subscriber: kzimmerman.

The impact on the A/B test has been resolved, but the underlying issue was not determined. See https://phabricator.wikimedia.org/T221197#5432076

Reassigning to @MNeisler to discuss with @ppelberg as they go through priorities.

Looks like this is a low priority task since it doesn't impact the A/B test; should we move to the Icebox or decline it?