Oversampling changes edit completion rate
Open, LowPublic
Actions

Assigned To

None

Authored By

	ppelberg
	Jul 13 2019, 12:40 AM

Description

Context

In T221197#5319337, we learned oversampling changes the edit completion rate between the two test groups: default-visual and default-source.

We do not currently know why this is.

Implications

If our randomization is not working properly (read: it is not truly random), the accuracy/legitimacy of all of our other of all of our test metrics is put into question.

"Done"

We have a reasonable explanation as to why oversampling changes the edit completion rate between the two test groups: default-visual and default-source.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T118063 Clean up the EditAttemptStep schema and its implementations
		Open		None	T227931 Oversampling changes edit completion rate

Event Timeline

ppelberg created this task.Jul 13 2019, 12:40 AM

ppelberg added a parent task: T221197: VE mobile default: A/B test post-deployment data checks.

ppelberg mentioned this in T221197: VE mobile default: A/B test post-deployment data checks.

JTannerWMF edited projects, added VisualEditor (Current work); removed VisualEditor.Jul 16 2019, 4:49 PM

nshahquinn-wmf triaged this task as High priority.Jul 30 2019, 1:20 PM

nshahquinn-wmf moved this task from Triage to Doing on the Product-Analytics board.

ppelberg moved this task from Incoming to In progress on the VisualEditor (Current work) board.Aug 9 2019, 4:52 PM

nshahquinn-wmf lowered the priority of this task from High to Medium.Aug 20 2019, 6:26 PM

nshahquinn-wmf removed a parent task: T221197: VE mobile default: A/B test post-deployment data checks.Aug 22 2019, 4:54 PM

nshahquinn-wmf added a parent task: T118063: Clean up the EditAttemptStep schema and its implementations.

The impact on the A/B test has been resolved, but the underlying issue was not determined. See https://phabricator.wikimedia.org/T221197#5432076

Reassigning to @MNeisler to discuss with @ppelberg as they go through priorities.

Looks like this is a low priority task since it doesn't impact the A/B test; should we move to the Icebox or decline it?

• mmodell edited projects, added Product-Analytics (Kanban); removed Product-Analytics.Oct 16 2019, 5:47 PM

• mmodell edited projects, added Product-Analytics; removed Product-Analytics (Kanban).Oct 16 2019, 5:51 PM

JTannerWMF edited projects, added Editing-team (Q3 2019-2020 Kanban Board); removed VisualEditor (Current work).Feb 5 2020, 12:06 AM

JTannerWMF moved this task from Needs Discussion / Investigation to Doing on the Editing-team (Q3 2019-2020 Kanban Board) board.

Restricted Application added a project: VisualEditor. · View Herald TranscriptFeb 5 2020, 12:06 AM

JTannerWMF edited projects, added Editing-team (Tracking); removed Editing-team (Q3 2019-2020 Kanban Board).Feb 5 2020, 1:00 AM

JTannerWMF moved this task from Backlog to Analytics on the Editing-team (Tracking) board.

MNeisler moved this task from Backlog to Icebox on the Product-Analytics board.Aug 12 2020, 7:41 PM

Removing task assignee due to inactivity, as this open task has been assigned for more than two years. See the email sent to the task assignee on February 06th 2022 (and T295729).

Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome.

If this task has been resolved in the meantime, or should not be worked on ("declined"), please update its task status via "Add Action… 🡒 Change Status".

Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator.

Oversampling changes edit completion rate Open, LowPublicActions