Page MenuHomePhabricator

Oversampling changes edit completion rate
Open, LowPublic

Description

Context

In T221197#5319337, we learned oversampling changes the edit completion rate between the two test groups: default-visual and default-source.

We do not currently know why this is.

Implications

If our randomization is not working properly (read: it is not truly random), the accuracy/legitimacy of all of our other of all of our test metrics is put into question.

"Done"

  • We have a reasonable explanation as to why oversampling changes the edit completion rate between the two test groups: default-visual and default-source.

Event Timeline

nshahquinn-wmf moved this task from Triage to Doing on the Product-Analytics board.
nshahquinn-wmf lowered the priority of this task from High to Medium.Aug 20 2019, 6:26 PM
kzimmerman lowered the priority of this task from Medium to Low.Aug 27 2019, 6:32 PM
kzimmerman moved this task from Doing to Backlog on the Product-Analytics board.
kzimmerman added a subscriber: kzimmerman.

The impact on the A/B test has been resolved, but the underlying issue was not determined. See https://phabricator.wikimedia.org/T221197#5432076

kzimmerman added subscribers: MNeisler, nshahquinn-wmf.

Reassigning to @MNeisler to discuss with @ppelberg as they go through priorities.

Looks like this is a low priority task since it doesn't impact the A/B test; should we move to the Icebox or decline it?