Maniphest T311972

[SPIKE] Investigate table of contents data and instrumentation
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ovasileva
	Jul 4 2022, 6:18 AM

Tags

Referenced Files

None

Subscribers

Description

Background

We are seeing unexpected results from the table of contents A/B test and would like to investigate whether and when there might be issues within our instrumentation (see T309682: Analyze table of contents A/B test for context)

Acceptance criteria

Review instrumentation for table of contents feature - look in particular for any possibilities for data loss, incorrect bucketing, or other issues
Review queries from T309682: Analyze table of contents A/B test along with @jwang

Related Objects

Mentioned In: T309682: Analyze table of contents A/B test
Mentioned Here: T309682: Analyze table of contents A/B test

Event Timeline

ovasileva created this task.Jul 4 2022, 6:18 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 4 2022, 6:18 AM

ovasileva triaged this task as High priority.Jul 4 2022, 6:18 AM

ovasileva mentioned this in T309682: Analyze table of contents A/B test.

LGoto assigned this task to bwang.Jul 5 2022, 5:11 PM

LGoto added a subscriber: jwang.

ovasileva updated the task description. (Show Details)Jul 5 2022, 5:14 PM

ovasileva updated the task description. (Show Details)

Jdlrobson moved this task from Incoming to Table of Contents on the Desktop Improvements (Vector 2022) board.Jul 8 2022, 6:16 PM

Queries and analysis in detail here: https://github.com/jenniferwang-wmf/WEB_table_of_contents/blob/master/analysis_4_AB_test.ipynb

Update: we found a bucketing issue, updated analysis and methodology here: https://nbviewer.org/github/jenniferwang-wmf/WEB_table_of_contents/blob/master/analysis_5_clicks_modeling.ipynb

bwang removed bwang as the assignee of this task.Jul 25 2022, 5:04 PM

bwang moved this task from Needs Analysis to Upcoming on the Web-Team-Backlog (Kanbanana-2022-23-Q1) board.

bwang subscribed.

ovasileva moved this task from Upcoming to Ready for Signoff on the Web-Team-Backlog (Kanbanana-2022-23-Q1) board.Jul 25 2022, 5:11 PM

LGoto assigned this task to ovasileva.Jul 25 2022, 5:20 PM

Results for a/b test are now complete. We can go ahead and resolve this task.